Re: [apache/incubator-mxnet] [RFC] Use TVMOp with GPU & Build without libcuda.so in CI (#18716)
I'm fine to disable tvm op (or mark it as experimental) for now, if it does need another 4 - 6 weeks to fully address the underlying problem, as we have some more urgent tasks on numpy side. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-mxnet/issues/18716#issuecomment-659071604
Re: [apache/incubator-mxnet] [RFC] A faster version of Gamma sampling on GPU. (#15928)
@ptrendx Thanks now I got what you mean. I'm open to what you proposed. while I think one of the major problems with the device api is the maintenance of the random generator (and it's states). -- You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-mxnet/issues/15928#issuecomment-522504256
Re: [apache/incubator-mxnet] [RFC] A faster version of Gamma sampling on GPU. (#15928)
@ptrendx If I understand correctly, "static assignment" is what current mxnet is doing, which is "ndarray on GPU" in @xidulu 's table. -- You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-mxnet/issues/15928#issuecomment-522219083
Re: [apache/incubator-mxnet] [RFC] A faster version of Gamma sampling on GPU. (#15928)
cc @apache/mxnet-committers I think we can gradually refactor current implementation (ndarray api) by adopting this new approach. @xidulu could you please fix the url links in your post. -- You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-mxnet/issues/15928#issuecomment-521931559
[RFC] Integrate TVM into Apache MXNet
Kindly remind people take a look at the posted RFC: https://github.com/apache/incubator-mxnet/issues/15465 and free free to leave your questions and suggestions -- Yizhi Liu Bay Area, the United States
Re: Apache MXNet Scala nightly-build now available on Apache Nexus
Thanks for attention. Here's an example demonstrates how to configure pom.xml to use the nightly build in your own project: https://github.com/yzhliu/mxnet-scala-example/blob/master/pom.xml#L152-L163 On Mon, Oct 29, 2018 at 2:53 PM Carin Meier wrote: > > Fantastic! This is going to be great for the Clojure MXNet users as well. > > On Mon, Oct 29, 2018 at 1:39 PM YiZhi Liu wrote: > > > Hi, > > > > > > > > I am happy to announce that the availability of the experimental > > nightly-build Scala package on Maven in Nexus. The nightly-builds, > > currently published as 1.3.1-SNAPSHOT version, include the latest changes > > from the master branch from apache/incubator-mxnet GitHub repo and refresh > > every day. Furthermore, we completely overhauled the MXNet library shipped > > inside and adopted the binary build logic used for Python pip, so now Scala > > users can also enjoy the portable MXNet with fully loaded features. > > Currently, the supported artifacts are: > > > > - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU) > > > > - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0) > > > > - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU) > > > > > > > > FAQ: > > > > - What's new in the nightly Scala packages compared to the previous Scala > > Maven artifacts? > > > > Most notably, better usability, more supported features, and better > > platform compatibility. The new packages now support Ubuntu 14.04 and > > Amazon Linux and support the same broad set of features as the existing pip > > packages. Besides, users are no longer required to install all the > > dependencies to use the nightly Scala packages as the included MXNet > > libraries are fully loaded. For the list of supported platforms and > > features, see [1]. > > > > > > > > - What's different in the build approach compared to what's in the previous > > Scala Maven packages? > > > > The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the > > same build commands that regular users would use when building from source > > [2]. This naive approach limits the compatibility to only similar > > environments as where the packages were built (which was Ubuntu 16.04 for > > Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more > > CentOS or Amazon Linux and may suffer the problem of mismatching of > > dependency versions due to the difference in their environment. > > > > On the other hand, Python users have long been enjoying portable and fully > > loaded MXNet packages from pip. The MXNet libraries in pip packages are > > built in more controlled environments with compatibility in mind. > > Dependencies for extended features are statically linked and masked so that > > no additional steps are needed. > > > > Now, in the new Scala nightly packages, the same approach as pip is used > > for building the MXNet library so that the Scala users can benefit from the > > same great features. > > > > > > > > - How do I get the new nightly packages? > > > > You can find the packages on Apache (Snapshots) repository [3] including > > pom files and jars, which you can download and add to your projects. > > Alternatively, you can configure your pom.xml to use the Maven repository. > > Remember to add repository config: > > > > > > > > > > > > Apache Snapshot > > > > https://repository.apache.org/content/groups/snapshots > > > > > > > > > > > > > > > > - What's the support level of the Scala nightly packages? > > > > This package is currently experimental, so use at your own risk. While the > > first nightly packages are already manually verified on several platforms, > > and the automated publish pipeline runs unit tests and integration tests on > > OSX and Ubuntu 14.04 before publishing, the testing is currently limited. > > > > > > > > - How can I help? > > > > We call on all MXNet Scala community to test out the new nightly packages. > > > > > > > > - I found a problem in using the package. What should I do? > > > > Please report this in the Github issue in [1]. > > > > > > > > - What's next? > > > > There are several more things to be done: > > > > 1. We are adding more comprehensive support for GPU versions for different > > CUDA versions. > > > > 2. We are adding more tests on more platforms. We are currently examining >
Apache MXNet Scala nightly-build now available on Apache Nexus
Hi, I am happy to announce that the availability of the experimental nightly-build Scala package on Maven in Nexus. The nightly-builds, currently published as 1.3.1-SNAPSHOT version, include the latest changes from the master branch from apache/incubator-mxnet GitHub repo and refresh every day. Furthermore, we completely overhauled the MXNet library shipped inside and adopted the binary build logic used for Python pip, so now Scala users can also enjoy the portable MXNet with fully loaded features. Currently, the supported artifacts are: - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU) - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0) - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU) FAQ: - What's new in the nightly Scala packages compared to the previous Scala Maven artifacts? Most notably, better usability, more supported features, and better platform compatibility. The new packages now support Ubuntu 14.04 and Amazon Linux and support the same broad set of features as the existing pip packages. Besides, users are no longer required to install all the dependencies to use the nightly Scala packages as the included MXNet libraries are fully loaded. For the list of supported platforms and features, see [1]. - What's different in the build approach compared to what's in the previous Scala Maven packages? The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the same build commands that regular users would use when building from source [2]. This naive approach limits the compatibility to only similar environments as where the packages were built (which was Ubuntu 16.04 for Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more CentOS or Amazon Linux and may suffer the problem of mismatching of dependency versions due to the difference in their environment. On the other hand, Python users have long been enjoying portable and fully loaded MXNet packages from pip. The MXNet libraries in pip packages are built in more controlled environments with compatibility in mind. Dependencies for extended features are statically linked and masked so that no additional steps are needed. Now, in the new Scala nightly packages, the same approach as pip is used for building the MXNet library so that the Scala users can benefit from the same great features. - How do I get the new nightly packages? You can find the packages on Apache (Snapshots) repository [3] including pom files and jars, which you can download and add to your projects. Alternatively, you can configure your pom.xml to use the Maven repository. Remember to add repository config: Apache Snapshot https://repository.apache.org/content/groups/snapshots - What's the support level of the Scala nightly packages? This package is currently experimental, so use at your own risk. While the first nightly packages are already manually verified on several platforms, and the automated publish pipeline runs unit tests and integration tests on OSX and Ubuntu 14.04 before publishing, the testing is currently limited. - How can I help? We call on all MXNet Scala community to test out the new nightly packages. - I found a problem in using the package. What should I do? Please report this in the Github issue in [1]. - What's next? There are several more things to be done: 1. We are adding more comprehensive support for GPU versions for different CUDA versions. 2. We are adding more tests on more platforms. We are currently examining the MXNet's Jenkins CI solution for supporting more platforms. 3. As we recognize that the build logic benefits the community, we plan to open source the build scripts in the mxnet main repo. See progress at [5]. Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this happen. Thanks to Sheng Zha for sharing the pip build logic, and for building the automated publishing pipelines. Special thanks to Qing and Zach for sharing their expertise in Scala with Sheng and guiding and supporting him locally for this milestone. [1] https://github.com/apache/incubator-mxnet/issues/8671 [2] https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105 [3] https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~ [4] https://maven.apache.org/repository-management.html [5] https://github.com/apache/incubator-mxnet/projects/13
Re: [Discussion] Recognise Reviewers, Besides Committers and PMC
+1 I also suggest add reviewer list link to the PR template, so that developers can easily request review from those reviewers. On Sun, Oct 21, 2018 at 8:30 PM Tianqi Chen wrote: > > I was suggesting something more concrete: > > - Add a Reviewers section to > https://github.com/apache/incubator-mxnet/blob/master/CONTRIBUTORS.md to > list a list of Reviewers. > - This is a "pesudo role", but holds weight as committers should highly > value their reviews during the PR process. > - The committers/PMC could actively look for good contributors and nominate > them as Reviewer. > - Contributors are encouraged to seek reviews from the list of reviewers. > - The committers should actively solicit code reviews from the reviewers > when reviewing PRs and take their reviews into serious consideration. > > - PMCs should actively look for new committers in the current Reviewers >- Notably, the history reviews plus contribution likely will provide a > good indication on whether the person can uphold the quality standard of > the codebase, and provide helpful feedbacks(which is the trait that needed > from committer to merge code) > > Tianqi > > > On Sun, Oct 21, 2018 at 5:13 PM Steffen Rochel > wrote: > > > +1 > > With the release announcement for MXNet 1.3 all contributors incl. code > > reviewers have been recognized. I suggest all future release announcements > > should include such recognition. Are you suggesting to highlight most > > active reviewers in release announcement or regularly (e.g. monthly), > > specifically from non-committers? > > > > On Sun, Oct 21, 2018 at 10:11 AM Tianqi Chen wrote: > > > > > Also re another email-thread(I sent out one with my institutional email > > > which get blocked initially, so this one was a bit duplication of that). > > I > > > think it should really be the job of committers to recognize potential > > > reviewers, github also makes it easier to do so, e.g. > > > > > > > > https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93&q=reviewed-by%3Apiiswrong > > > > > > Tianqi > > > > > > On Fri, Oct 19, 2018 at 12:05 PM Carin Meier > > wrote: > > > > > > > +1 Great idea. Adding a name to the contributor list is a good idea. > > > Also, > > > > I've found that thanking the person for the review on the PR is another > > > way > > > > to express gratitude for their time and effort. > > > > > > > > On Fri, Oct 19, 2018 at 2:51 PM Tianqi Chen wrote: > > > > > > > > > Dear MXNet Community: > > > > > > > > > > There is a great discussion going on in terms of lowering the barrier > > > of > > > > > entries and encourage more contribution to the project. One of the > > > > general > > > > > goals is to encourage a broader pool of contributions. I want to make > > > the > > > > > following proposal: > > > > > > > > > > Besides Committers and PMC, let us also recognize Reviewers in the > > > > > community. This is a "pseudo role" as there is no such official role > > > in > > > > > Apache. But I want to explore the possibility of recognizing active > > > > > reviewers for example, by adding a list of names in the contributor > > > list. > > > > > In general, I find it is really helpful to have more code reviews. > > > > > Recognizing good reviewers early enables us to find committer > > > candidates, > > > > > and encourage them to contribute and understand what is the bar of > > code > > > > > quality that is required to merge the code. > > > > > > > > > > This can provide the community with more evidence when recruiting new > > > > > committers. After all the write access of committership is about to > > the > > > > > code and understand the consequence of the responsibility -- which is > > > > > usually can be found in high-quality review history. > > > > > > > > > > Please let me know what you think. > > > > > > > > > > Tianqi > > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Feedback request for new Java API
Yes agreement and disagreement stay at technical level only:) Back to the problem, they are unnecessary but good in terms of, 1. Still not good for java users to write 3 nulls in a function call with 5 or 4 args 2. Every function call with a “tail” null for arg “out”. I would say, makes it seems not a serious api design to our users 3. Users have uniformed experience, nothing surprising. Given the reasons I listed before, I don’t see things bad for this. I agree we can vote. I suggest to have two votes, one for builder, one for separating java and scala objects. On Sat, Sep 29, 2018 at 7:43 PM Naveen Swamy wrote: > Ah! we agree on something :) lets get more opinions, I am happy to go with > it. > > On Sat, Sep 29, 2018 at 10:40 PM YiZhi Liu wrote: > > > Also sometimes people may not be at the same page when talking about > option > > #2. What I insist is the builder classes for each operator. Otherwise I > > actually more support Naveen’s approach - not to totally separate java > and > > scala objects. > > > > On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > > > > > No you haven't answered my question "Since you agree to have 30+ > > > operators have Builder, what prevents from > > > having all of them have Builder?" > > > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy > wrote: > > > > > > > > I think we have had enough of an debate between the two of us and I > > have > > > > already listed my reasons, I will stop here and see what others say > > > given > > > > my reasoning. > > > > > > > > -1 to #2) > > > > > > > > Also, by lecture I meant to say "I don't want to list all the > problems > > > > with unnecessary complications and talk about how to design software" > > > > > > > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu > > wrote: > > > > > > > > > And if we find incorrect declaration, we fix it, not simply > assuming > > > > > many of them also has problem and we cannot rely on them - > otherwise > > > > > the type-safe APIs in Scala also does not make sense. > > > > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu > > wrote: > > > > > > > > > > > > It also makes sense to me if we have it under namespace NDArray, > > not > > > > > > creating new JavaNDArray. But again, uniform experience is > > important. > > > > > > > > > > > > What I responded is your comment "keep scala macros minimum", I > > don't > > > > > > think "scala macro" equals "cryptic code". Even though it does, > > what > > > > > > we need to do is to find an alternative way to do code > generation, > > > not > > > > > > making code generation minimum. > > > > > > > > > > > > Since you agree to have 30+ operators have Builder, what prevents > > > from > > > > > > having all of them have Builder? > > > > > > - They're auto-generated, the auto-generation "cryptic" code is > > > anyway > > > > > > there. And "two different paths of code" (though I don't totally > > > > > > agree) is anyway there. > > > > > > - What else? 200+ classes is a very tiny increasing in file size > > > > > > (~3MB) compare to current status. And won't have any performance > > > issue > > > > > > on modern JVM. > > > > > > > > > > > > Just remind, technical discussion is not about who gives who a > > > lecture. > > > > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy > > > > wrote: > > > > > > > > > > > > > > Well, I am not sure(I don't think) we need Builder for every > API > > in > > > > > > > NDArray. For APIs that take long list of parameters, I agree to > > add > > > > > Builder. > > > > > > > Look at the API distribution based on number of arguments here: > > > > > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > > > > > > > about 30 APIs have 7 or more arguments.. I agree to add > Builders > > > for > > > > > these > > > > > > > APIs not separately but to the existing Scala APIs but not > > > separately > > > > > only >
Re: Feedback request for new Java API
Also sometimes people may not be at the same page when talking about option #2. What I insist is the builder classes for each operator. Otherwise I actually more support Naveen’s approach - not to totally separate java and scala objects. On Sat, Sep 29, 2018 at 7:35 PM YiZhi Liu wrote: > No you haven't answered my question "Since you agree to have 30+ > operators have Builder, what prevents from > having all of them have Builder?" > On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy wrote: > > > > I think we have had enough of an debate between the two of us and I have > > already listed my reasons, I will stop here and see what others say > given > > my reasoning. > > > > -1 to #2) > > > > Also, by lecture I meant to say "I don't want to list all the problems > > with unnecessary complications and talk about how to design software" > > > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote: > > > > > And if we find incorrect declaration, we fix it, not simply assuming > > > many of them also has problem and we cannot rely on them - otherwise > > > the type-safe APIs in Scala also does not make sense. > > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote: > > > > > > > > It also makes sense to me if we have it under namespace NDArray, not > > > > creating new JavaNDArray. But again, uniform experience is important. > > > > > > > > What I responded is your comment "keep scala macros minimum", I don't > > > > think "scala macro" equals "cryptic code". Even though it does, what > > > > we need to do is to find an alternative way to do code generation, > not > > > > making code generation minimum. > > > > > > > > Since you agree to have 30+ operators have Builder, what prevents > from > > > > having all of them have Builder? > > > > - They're auto-generated, the auto-generation "cryptic" code is > anyway > > > > there. And "two different paths of code" (though I don't totally > > > > agree) is anyway there. > > > > - What else? 200+ classes is a very tiny increasing in file size > > > > (~3MB) compare to current status. And won't have any performance > issue > > > > on modern JVM. > > > > > > > > Just remind, technical discussion is not about who gives who a > lecture. > > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy > wrote: > > > > > > > > > > Well, I am not sure(I don't think) we need Builder for every API in > > > > > NDArray. For APIs that take long list of parameters, I agree to add > > > Builder. > > > > > Look at the API distribution based on number of arguments here: > > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > > > > > about 30 APIs have 7 or more arguments.. I agree to add Builders > for > > > these > > > > > APIs not separately but to the existing Scala APIs but not > separately > > > only > > > > > for Java. > > > > > APIs sorted by number of arguments is here, take a look : > > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5 > > > > > > > > > > Many of the arguments i think are actually mandatory but > incorrectly > > > > > declared optional on the backend, for example look at SwapAxis > > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 : > > > > > Option[Int] = None, out : Option[NDArray] = None) : > NDArrayFuncReturn" > > > > > Why is dim1 and dim2 Optional, this is an error in the declaration > on > > > the > > > > > backend, I think there might be many of these? > > > > > > > > > > My answers to your other responses are below inline: > > > > > > > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu > wrote: > > > > > > > > > > > Some of my comments inline: > > > > > > > > > > > > > Why can we not create the builder just for these APIs( which we > > > > > > discussed), why is it necessary to add 200 Apis > > > > > > It is about unified user-experience. And we get rid of annoying > extra > > > > > > "out=null" in every operator. > > > > > > > > > > > > > Are you suggesting to crea
Re: Feedback request for new Java API
No you haven't answered my question "Since you agree to have 30+ operators have Builder, what prevents from having all of them have Builder?" On Sat, Sep 29, 2018 at 7:30 PM Naveen Swamy wrote: > > I think we have had enough of an debate between the two of us and I have > already listed my reasons, I will stop here and see what others say given > my reasoning. > > -1 to #2) > > Also, by lecture I meant to say "I don't want to list all the problems > with unnecessary complications and talk about how to design software" > > On Sat, Sep 29, 2018 at 10:15 PM YiZhi Liu wrote: > > > And if we find incorrect declaration, we fix it, not simply assuming > > many of them also has problem and we cannot rely on them - otherwise > > the type-safe APIs in Scala also does not make sense. > > On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote: > > > > > > It also makes sense to me if we have it under namespace NDArray, not > > > creating new JavaNDArray. But again, uniform experience is important. > > > > > > What I responded is your comment "keep scala macros minimum", I don't > > > think "scala macro" equals "cryptic code". Even though it does, what > > > we need to do is to find an alternative way to do code generation, not > > > making code generation minimum. > > > > > > Since you agree to have 30+ operators have Builder, what prevents from > > > having all of them have Builder? > > > - They're auto-generated, the auto-generation "cryptic" code is anyway > > > there. And "two different paths of code" (though I don't totally > > > agree) is anyway there. > > > - What else? 200+ classes is a very tiny increasing in file size > > > (~3MB) compare to current status. And won't have any performance issue > > > on modern JVM. > > > > > > Just remind, technical discussion is not about who gives who a lecture. > > > On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote: > > > > > > > > Well, I am not sure(I don't think) we need Builder for every API in > > > > NDArray. For APIs that take long list of parameters, I agree to add > > Builder. > > > > Look at the API distribution based on number of arguments here: > > > > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb > > > > about 30 APIs have 7 or more arguments.. I agree to add Builders for > > these > > > > APIs not separately but to the existing Scala APIs but not separately > > only > > > > for Java. > > > > APIs sorted by number of arguments is here, take a look : > > > > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5 > > > > > > > > Many of the arguments i think are actually mandatory but incorrectly > > > > declared optional on the backend, for example look at SwapAxis > > > > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 : > > > > Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn" > > > > Why is dim1 and dim2 Optional, this is an error in the declaration on > > the > > > > backend, I think there might be many of these? > > > > > > > > My answers to your other responses are below inline: > > > > > > > > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote: > > > > > > > > > Some of my comments inline: > > > > > > > > > > > Why can we not create the builder just for these APIs( which we > > > > > discussed), why is it necessary to add 200 Apis > > > > > It is about unified user-experience. And we get rid of annoying extra > > > > > "out=null" in every operator. > > > > > > > > > > > Are you suggesting to create builder for each and every API? > > > > > Only for those are necessary. For NDArray.XXX, yes. > > > > > > > > > I think this is a ridiculous list of Builders, I think we can keep the > > > > 'out' parameter > > > > > > > > > 1) The NDArray APIs in question are not following functional style of > > > > > programming, in fact they are just static methods defined on an > > > > > NDArray object - so Scala users are not losing much by using null in > > > > > place of None. > > > > > You can create a implicit to maintain backward compatibility > > > > > - I doubt i
Re: Feedback request for new Java API
And if we find incorrect declaration, we fix it, not simply assuming
many of them also has problem and we cannot rely on them - otherwise
the type-safe APIs in Scala also does not make sense.
On Sat, Sep 29, 2018 at 7:10 PM YiZhi Liu wrote:
>
> It also makes sense to me if we have it under namespace NDArray, not
> creating new JavaNDArray. But again, uniform experience is important.
>
> What I responded is your comment "keep scala macros minimum", I don't
> think "scala macro" equals "cryptic code". Even though it does, what
> we need to do is to find an alternative way to do code generation, not
> making code generation minimum.
>
> Since you agree to have 30+ operators have Builder, what prevents from
> having all of them have Builder?
> - They're auto-generated, the auto-generation "cryptic" code is anyway
> there. And "two different paths of code" (though I don't totally
> agree) is anyway there.
> - What else? 200+ classes is a very tiny increasing in file size
> (~3MB) compare to current status. And won't have any performance issue
> on modern JVM.
>
> Just remind, technical discussion is not about who gives who a lecture.
> On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote:
> >
> > Well, I am not sure(I don't think) we need Builder for every API in
> > NDArray. For APIs that take long list of parameters, I agree to add Builder.
> > Look at the API distribution based on number of arguments here:
> > https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> > about 30 APIs have 7 or more arguments.. I agree to add Builders for these
> > APIs not separately but to the existing Scala APIs but not separately only
> > for Java.
> > APIs sorted by number of arguments is here, take a look :
> > https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
> >
> > Many of the arguments i think are actually mandatory but incorrectly
> > declared optional on the backend, for example look at SwapAxis
> > "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> > Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
> > Why is dim1 and dim2 Optional, this is an error in the declaration on the
> > backend, I think there might be many of these?
> >
> > My answers to your other responses are below inline:
> >
> > On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
> >
> > > Some of my comments inline:
> > >
> > > > Why can we not create the builder just for these APIs( which we
> > > discussed), why is it necessary to add 200 Apis
> > > It is about unified user-experience. And we get rid of annoying extra
> > > "out=null" in every operator.
> > >
> > > > Are you suggesting to create builder for each and every API?
> > > Only for those are necessary. For NDArray.XXX, yes.
> > >
> > I think this is a ridiculous list of Builders, I think we can keep the
> > 'out' parameter
> >
> > > 1) The NDArray APIs in question are not following functional style of
> > > programming, in fact they are just static methods defined on an
> > > NDArray object - so Scala users are not losing much by using null in
> > > place of None.
> > > You can create a implicit to maintain backward compatibility
> > > - I doubt implicit can work in such case from None -> null.
> > >
> >
> > It is just writing getOrElse in your implicit, so it will work.
> > scala> implicit def optionStringToString(a: Option[String]): String = {
> > | a.getOrElse(null)
> > | }
> >
> > 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone
> > > - As I explained how it can improve user experiences
> > >
> > I don't think we need to write builders for 221 APIs we have, may be for 30
> > or so. Uniform experience is good goal but it also has to be practical and
> > make sense.
> >
> > 3) this is adding another 100s of APIs unnecessarily, we are starting with
> > > NDArray but we can't stop there, we will have to do this for Symbol,
> > > Executor, Iterators, etc., .
> > > - This is a good point, actually I prefer not to make JavaExecutor,
> > > JavaIterators
> > >
> > What I was aiming is also users have the same experience across Interfaces
> > - now you are forgoing uniform experience, so like you said its all
> > trade-off and a good trade-off doesn't cause too much overhead/
> >
> >
> > > 4) I don't want to be fixing bugs and maint
Re: Feedback request for new Java API
It also makes sense to me if we have it under namespace NDArray, not
creating new JavaNDArray. But again, uniform experience is important.
What I responded is your comment "keep scala macros minimum", I don't
think "scala macro" equals "cryptic code". Even though it does, what
we need to do is to find an alternative way to do code generation, not
making code generation minimum.
Since you agree to have 30+ operators have Builder, what prevents from
having all of them have Builder?
- They're auto-generated, the auto-generation "cryptic" code is anyway
there. And "two different paths of code" (though I don't totally
agree) is anyway there.
- What else? 200+ classes is a very tiny increasing in file size
(~3MB) compare to current status. And won't have any performance issue
on modern JVM.
Just remind, technical discussion is not about who gives who a lecture.
On Sat, Sep 29, 2018 at 6:41 PM Naveen Swamy wrote:
>
> Well, I am not sure(I don't think) we need Builder for every API in
> NDArray. For APIs that take long list of parameters, I agree to add Builder.
> Look at the API distribution based on number of arguments here:
> https://gist.github.com/nswamy/2dea72e514cc7bfc675f68aef9fe78bb
> about 30 APIs have 7 or more arguments.. I agree to add Builders for these
> APIs not separately but to the existing Scala APIs but not separately only
> for Java.
> APIs sorted by number of arguments is here, take a look :
> https://gist.github.com/nswamy/e941cb94658b3960eec40bf00b970ac5
>
> Many of the arguments i think are actually mandatory but incorrectly
> declared optional on the backend, for example look at SwapAxis
> "def SwapAxis (data : NDArray, dim1 : Option[Int] = None, dim2 :
> Option[Int] = None, out : Option[NDArray] = None) : NDArrayFuncReturn"
> Why is dim1 and dim2 Optional, this is an error in the declaration on the
> backend, I think there might be many of these?
>
> My answers to your other responses are below inline:
>
> On Sat, Sep 29, 2018 at 3:37 PM YiZhi Liu wrote:
>
> > Some of my comments inline:
> >
> > > Why can we not create the builder just for these APIs( which we
> > discussed), why is it necessary to add 200 Apis
> > It is about unified user-experience. And we get rid of annoying extra
> > "out=null" in every operator.
> >
> > > Are you suggesting to create builder for each and every API?
> > Only for those are necessary. For NDArray.XXX, yes.
> >
> I think this is a ridiculous list of Builders, I think we can keep the
> 'out' parameter
>
> > 1) The NDArray APIs in question are not following functional style of
> > programming, in fact they are just static methods defined on an
> > NDArray object - so Scala users are not losing much by using null in
> > place of None.
> > You can create a implicit to maintain backward compatibility
> > - I doubt implicit can work in such case from None -> null.
> >
>
> It is just writing getOrElse in your implicit, so it will work.
> scala> implicit def optionStringToString(a: Option[String]): String = {
> | a.getOrElse(null)
> | }
>
> 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone
> > - As I explained how it can improve user experiences
> >
> I don't think we need to write builders for 221 APIs we have, may be for 30
> or so. Uniform experience is good goal but it also has to be practical and
> make sense.
>
> 3) this is adding another 100s of APIs unnecessarily, we are starting with
> > NDArray but we can't stop there, we will have to do this for Symbol,
> > Executor, Iterators, etc., .
> > - This is a good point, actually I prefer not to make JavaExecutor,
> > JavaIterators
> >
> What I was aiming is also users have the same experience across Interfaces
> - now you are forgoing uniform experience, so like you said its all
> trade-off and a good trade-off doesn't cause too much overhead/
>
>
> > 4) I don't want to be fixing bugs and maintaining code in 2 places.
> > - Type-safe parsing is shared. I think Qing is more qualified to comment.
>
> It creates two different paths of code for Scala and Java - how is it going
> to be shared. I am afraid we are going to make it more complicated than
> necessary by duplicating code.
>
> >
>
> 5) I want the cryptic code(# scala macros) to a minimum.
> > - MXNet decides to do operator generation in frontend bindings. It's
> > the developers' responsibility to understand the techniques they are
> > using. Maybe not a so proper analogy - "I don't know RL / RL is hard
> > to tune / ..." i
Re: Time out for Travis CI
This makes sense. Thanks On Sat, Sep 29, 2018 at 6:36 PM kellen sunderland < kellen.sunderl...@gmail.com> wrote: > Hey Zhennan, yes this is the exact problem, and I agree with your points > completely. This is why when we first added Travis we attempted to > communicate that it would be informational only, and that we'd need to > iterate on the config before it would be a test that people should consider > 'required'. Apologies, we should have been more straightforward about > those tradeoffs. The strong point in favour of adding Travis in > informational mode was that we had a serious MacOS specific bug that we > wanted to verify was fixed. > > The good news is I've opened a PR which I hope will speed up these builds > to the point that they won't rely on caching. Once it is merged it would > be very helpful if you could rebase on this PR and test to ensure that > large changes no longer hit the global timeout without cache. > https://github.com/apache/incubator-mxnet/pull/12706 > > On Sun, Sep 30, 2018 at 2:48 AM Qin, Zhennan > wrote: > > > Hi YiZhi and Kellen, > > > > From my point of view, travis should be able to get passed from a scratch > > build. Pending result on ccache hit/miss is not a good idea. For this PR, > > as it changed many header file, lots of files need be recompiled, just > like > > a scratch build. I think that's the reason that travis timeout. This > should > > be fixed before enabling travis, as it will block any change to those > base > > header file. Again, it's not a special case with this PR only, you can > find > > same problem on other PRs: > > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_source=github_status&utm_medium=notification > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/434404305?utm_source=github_status&utm_medium=notification > > > > > > Thanks, > > Zhennan > > > > -Original Message- > > From: YiZhi Liu [mailto:eazhi....@gmail.com] > > Sent: Sunday, September 30, 2018 5:15 AM > > To: eazhi@gmail.com > > Cc: dev@mxnet.incubator.apache.org > > Subject: Re: Time out for Travis CI > > > > while other PRs are all good. > > On Sat, Sep 29, 2018 at 2:13 PM YiZhi Liu wrote: > > > > > > Honestly I don't know yet. I can help to investigate. Just given the > > > evidence that, travis timeout every time it gets re-triggered - 2 > > > times at least. Correct me if I'm wrong @ Zhennan On Sat, Sep 29, 2018 > > > at 1:54 PM kellen sunderland wrote: > > > > > > > > Reading over the PR I don't see what aspects would cause extra > > > > runtime YiZhi, could you point them out? > > > > > > > > On Sat, Sep 29, 2018 at 8:46 PM YiZhi Liu > wrote: > > > > > > > > > Kellen, I think this PR introduces extra runtime in CI, thus > > > > > causes the timeout. Which means, once merged, every PR later will > > > > > see same timeout in travis. > > > > > > > > > > So shall we modify the changes to decrease the test running time? > > > > > or just disable the Travis CI? > > > > > > > > > > > > > > > On Fri, Sep 28, 2018 at 9:17 PM Qin, Zhennan > > > > > > > > > > wrote: > > > > > > > > > > > > Hi Kellen, > > > > > > > > > > > > Thanks for your explanation. Do you have a time plan to solve > > > > > > the > > > > > timeout issue? Rebasing can't work for my case. Or shall we run it > > > > > silently to disallow it voting X for overall CI result? Because > > > > > most developers are used to ignore the PRs with 'X'. > > > > > > > > > > > > Thanks, > > > > > > Zhennan > > > > > > > > > > > > -Original Message- > > > > > > From: kellen sunderland [mailto:kellen.sunderl...@gmail.com] > > > > > > Sent: Friday, September 28, 2018 10:38 PM > > > > > > To: dev@mxnet.incubator.apache.org > > > > > > Subject: Re: Time out for Travis CI > > > > > > > > > > > > Hey Zhennan, you're safe to ignore Travis failures for now. > > > > > > They're > > > > > just informational. > > > > > > > > > > > > The reason you sometimes see quick builds and
Re: Time out for Travis CI
while other PRs are all good. On Sat, Sep 29, 2018 at 2:13 PM YiZhi Liu wrote: > > Honestly I don't know yet. I can help to investigate. Just given the > evidence that, travis timeout every time it gets re-triggered - 2 > times at least. Correct me if I'm wrong @ Zhennan > On Sat, Sep 29, 2018 at 1:54 PM kellen sunderland > wrote: > > > > Reading over the PR I don't see what aspects would cause extra runtime > > YiZhi, could you point them out? > > > > On Sat, Sep 29, 2018 at 8:46 PM YiZhi Liu wrote: > > > > > Kellen, I think this PR introduces extra runtime in CI, thus causes > > > the timeout. Which means, once merged, every PR later will see same > > > timeout in travis. > > > > > > So shall we modify the changes to decrease the test running time? or > > > just disable the Travis CI? > > > > > > > > > On Fri, Sep 28, 2018 at 9:17 PM Qin, Zhennan > > > wrote: > > > > > > > > Hi Kellen, > > > > > > > > Thanks for your explanation. Do you have a time plan to solve the > > > timeout issue? Rebasing can't work for my case. Or shall we run it > > > silently > > > to disallow it voting X for overall CI result? Because most developers are > > > used to ignore the PRs with 'X'. > > > > > > > > Thanks, > > > > Zhennan > > > > > > > > -Original Message- > > > > From: kellen sunderland [mailto:kellen.sunderl...@gmail.com] > > > > Sent: Friday, September 28, 2018 10:38 PM > > > > To: dev@mxnet.incubator.apache.org > > > > Subject: Re: Time out for Travis CI > > > > > > > > Hey Zhennan, you're safe to ignore Travis failures for now. They're > > > just informational. > > > > > > > > The reason you sometimes see quick builds and sometimes see slow builds > > > is that we're making use of ccache in between builds. If your PR is > > > similar to what's in master you should build very quickly, if not it's > > > going to take a while and likely time out. If you see timeouts rebasing > > > may speed things up. Unfortunately the timeouts are global and we're not > > > able to increase them. I'm hoping that adding artifact caching will speed > > > up future builds to the point that test runs and builds can be executed in > > > under the global limit (which is ~50 minutes). > > > > > > > > -Kellen > > > > > > > > > > > > On Fri, Sep 28, 2018 at 4:05 PM Qin, Zhennan > > > wrote: > > > > > > > > > Hi MXNet devs, > > > > > > > > > > I'm struggled with new Travis CI for a while, it always run time out > > > > > for this PR: > > > > > https://github.com/apache/incubator-mxnet/pull/12530 > > > > > > > > > > Most of the time, Jenkins CI can pass, while Travis can't be finished > > > > > within 50 minutes. For this PR, it shouldn't affect much on the build > > > > > time or unit test time. Also, I saw other PR has same problem, eg. > > > > > > > > > > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_sour > > > > > ce=github_status&utm_medium=notification > > > > > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/434404305?utm_sour > > > > > ce=github_status&utm_medium=notification > > > > > > > > > > According to the time stamp from Travis, all passed PR are within > > > > > small code change, and can complete `make -j2` within 25s. But for > > > > > timeout case, 'make -j2' will need about 1600s. Does Travis do > > > > > incremental build for each test? Shall we increase time limit for > > > > > large PR? Can we add more time stamp for build and unites stage to > > > help understand what's going on there? > > > > > > > > > > Thanks in advance, > > > > > Zhennan > > > > > > > > > > > > > > > > > -- > > > Yizhi Liu > > > DMLC member > > > Amazon Web Services > > > Vancouver, Canada > > > > > > > -- > Yizhi Liu > DMLC member > Amazon Web Services > Vancouver, Canada -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Time out for Travis CI
Honestly I don't know yet. I can help to investigate. Just given the evidence that, travis timeout every time it gets re-triggered - 2 times at least. Correct me if I'm wrong @ Zhennan On Sat, Sep 29, 2018 at 1:54 PM kellen sunderland wrote: > > Reading over the PR I don't see what aspects would cause extra runtime > YiZhi, could you point them out? > > On Sat, Sep 29, 2018 at 8:46 PM YiZhi Liu wrote: > > > Kellen, I think this PR introduces extra runtime in CI, thus causes > > the timeout. Which means, once merged, every PR later will see same > > timeout in travis. > > > > So shall we modify the changes to decrease the test running time? or > > just disable the Travis CI? > > > > > > On Fri, Sep 28, 2018 at 9:17 PM Qin, Zhennan > > wrote: > > > > > > Hi Kellen, > > > > > > Thanks for your explanation. Do you have a time plan to solve the > > timeout issue? Rebasing can't work for my case. Or shall we run it silently > > to disallow it voting X for overall CI result? Because most developers are > > used to ignore the PRs with 'X'. > > > > > > Thanks, > > > Zhennan > > > > > > -Original Message- > > > From: kellen sunderland [mailto:kellen.sunderl...@gmail.com] > > > Sent: Friday, September 28, 2018 10:38 PM > > > To: dev@mxnet.incubator.apache.org > > > Subject: Re: Time out for Travis CI > > > > > > Hey Zhennan, you're safe to ignore Travis failures for now. They're > > just informational. > > > > > > The reason you sometimes see quick builds and sometimes see slow builds > > is that we're making use of ccache in between builds. If your PR is > > similar to what's in master you should build very quickly, if not it's > > going to take a while and likely time out. If you see timeouts rebasing > > may speed things up. Unfortunately the timeouts are global and we're not > > able to increase them. I'm hoping that adding artifact caching will speed > > up future builds to the point that test runs and builds can be executed in > > under the global limit (which is ~50 minutes). > > > > > > -Kellen > > > > > > > > > On Fri, Sep 28, 2018 at 4:05 PM Qin, Zhennan > > wrote: > > > > > > > Hi MXNet devs, > > > > > > > > I'm struggled with new Travis CI for a while, it always run time out > > > > for this PR: > > > > https://github.com/apache/incubator-mxnet/pull/12530 > > > > > > > > Most of the time, Jenkins CI can pass, while Travis can't be finished > > > > within 50 minutes. For this PR, it shouldn't affect much on the build > > > > time or unit test time. Also, I saw other PR has same problem, eg. > > > > > > > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_sour > > > > ce=github_status&utm_medium=notification > > > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/434404305?utm_sour > > > > ce=github_status&utm_medium=notification > > > > > > > > According to the time stamp from Travis, all passed PR are within > > > > small code change, and can complete `make -j2` within 25s. But for > > > > timeout case, 'make -j2' will need about 1600s. Does Travis do > > > > incremental build for each test? Shall we increase time limit for > > > > large PR? Can we add more time stamp for build and unites stage to > > help understand what's going on there? > > > > > > > > Thanks in advance, > > > > Zhennan > > > > > > > > > > > > -- > > Yizhi Liu > > DMLC member > > Amazon Web Services > > Vancouver, Canada > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Feedback request for new Java API
Some of my comments inline: > Why can we not create the builder just for these APIs( which we discussed), > why is it necessary to add 200 Apis It is about unified user-experience. And we get rid of annoying extra "out=null" in every operator. > Are you suggesting to create builder for each and every API? Only for those are necessary. For NDArray.XXX, yes. 1) The NDArray APIs in question are not following functional style of programming, in fact they are just static methods defined on an NDArray object - so Scala users are not losing much by using null in place of None. You can create a implicit to maintain backward compatibility - I doubt implicit can work in such case from None -> null. 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone - As I explained how it can improve user experiences 3) this is adding another 100s of APIs unnecessarily, we are starting with NDArray but we can't stop there, we will have to do this for Symbol, Executor, Iterators, etc., . - This is a good point, actually I prefer not to make JavaExecutor, JavaIterators 4) I don't want to be fixing bugs and maintaining code in 2 places. - Type-safe parsing is shared. I think Qing is more qualified to comment. 5) I want the cryptic code(# scala macros) to a minimum. - MXNet decides to do operator generation in frontend bindings. It's the developers' responsibility to understand the techniques they are using. Maybe not a so proper analogy - "I don't know RL / RL is hard to tune / ..." is not a reason for "I want to keep RL implementation in MXNet as a small part as possible" 6) increased compilation time & bad developer experience - the time to compile has gone up quite a bit since we added the APIs last release on my 3 year old laptop already.. I think adding 400+ APIs unnecessarily would significantly increase build time and bad developer experience - I don't think increasing such a bit compilation time is a problem compared to bad user experience. 7) I want to keep the core of the framework to be in Scala - because it allows you to write concise code - Yes it has a bit of learning curve, not everyone needs to know. I would rather invest in solidifying the Scala APIs and add more features in Scala(RNN, Support GluonHybridizedBlock...there is quite bit of work ) - do you want to rewrite everything in Scala and Java. - I agree with "don't rewrite everything in Scala and Java", IMO JavaNDArray is the only one good to have. JShape, JContext, etc. are not so necessary. 8) Also, the discussion is not creating NDArray class for Java, just generate certain APIs to cater for Java incompatibility. - Yes I agree it's about "generate certain APIs to cater for Java incompatibility", though I think NDArray.api.XXX does not meet Java users' demands. On Sat, Sep 29, 2018 at 12:05 PM Naveen Swamy wrote: > > I know it is about trade-off. I am suggesting a trade-off , how many apis do > we have that takes too many parameters ? > From what I recall its around 20. Why can we not create the builder just for > these APIs( which we discussed), why is it necessary to add 200 Apis ? > Are you suggesting to create builder for each and every API? > > I disagree with your opinion that they are not important and would like to > hear from others. > > I am curious to see how the #2 looks like compared to #1 > Andrew/Qing, can you paste the generated Apis that you have for both Scala > and Java in a gist please. > > > On Sep 29, 2018, at 2:41 PM, YiZhi Liu wrote: > > > > Naveen, software designing is all about tradeoff, every feature we > > introduce causes more compiling time, more efforts to maintain, etc. > > > > The main difference is. > > > > Option #1: Java users do > > NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, > > null, null, null, null, null, null); > > (and because every operator has an argument "out", users need to add > > an extra "null" to the function call almost every time.) > > > > Option #2, Java users do > > JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > > > > I don't think any of the reasons you listed is so important as the > > benefit above we got from option #2. > >> On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: > >> > >> Java APIs are not like Clojure - The current proposal is only to build a > >> few thin wrappers for Inference. > >> > >> To better represent the two cases and this discussion in particular, here > >> is an example API > >> > >> 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > >> : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncRetu
Re: Feedback request for new Java API
Moreover, regarding "add 200+ APIs" - this was exactly what we did for type-safe APIs in Scala, we have NDArray/Symbol.BatchNorm and NDArray/Symbol.api.BatchNorm - why did we decide to do that? Because we thought type-safe could provide better user-experience. Given the example I listed above, I think such user-experience benefits are even greater than what we got from type-safe. On Sat, Sep 29, 2018 at 11:41 AM YiZhi Liu wrote: > > Naveen, software designing is all about tradeoff, every feature we > introduce causes more compiling time, more efforts to maintain, etc. > > The main difference is. > > Option #1: Java users do > NDArray.BatchNorm(data, gamma, beta, null, null, null, null, null, > null, null, null, null, null, null); > (and because every operator has an argument "out", users need to add > an extra "null" to the function call almost every time.) > > Option #2, Java users do > JavaNDArray.BatchNorm(data).setGamma(gamma).setBeta(beta).invoke(); > > I don't think any of the reasons you listed is so important as the > benefit above we got from option #2. > On Sat, Sep 29, 2018 at 8:24 AM Naveen Swamy wrote: > > > > Java APIs are not like Clojure - The current proposal is only to build a > > few thin wrappers for Inference. > > > > To better represent the two cases and this discussion in particular, here > > is an example API > > > > 1) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : Option[NDArray] = None) : org.apache.mxnet.NDArrayFuncReturn > > or > > 2) def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : NDArray) : org.apache.mxnet.NDArrayFuncReturn > > > > The discussion is should we add(generate) 200+ APIs to make it Java > > compatible, ie., remove the Option class and the None default value which > > Java does not understand from Option 1) > > > > my suggestion was to remove the Option class and create a implicit for > > backward compatibility and use null instead of None, Andrew and I disagreed > > on this, so I suggested to raise a discussion on dev@ to get more opinions > > and one of us will disagree and commit. Thanks for raising it :) > > > > | * def Activation (data : org.apache.mxnet.NDArray, act_type : String, out > > : NDArray = null) : org.apache.mxnet.NDArrayFuncReturn | > > -- > > > > 1) It is not true that Scala users will lose *default/optional* arguments - > > if we followed the above, they will use null or None, though I do not like > > using nulls, this is a fine compromise. > > To keep backward compatibility we can create a implicit to convert > > Option.None to nulls and Option.Some-> Option.get(), so you are not going > > to break users who might have been using the APIs that were released in > > 1.3. The current incompatibility is only this w.r.t. NDArrays. > > > > 2) Now about the Scala Macros - they are not simple to read or use, When I > > and Qing started working on the #Scala Macros to improve the APIs, it took > > us a good amount of time to get a hang of it. I don't want to add > > additional code when not necessary. > > > > My suggestion and vote is to modify existing Macro(i.e., #1 from the > > original email with the necessary clarification above) and make it > > compatible with Java > > Here are my reasons > > 1) The NDArray APIs in question are not following functional style of > > programming, in fact they are just static methods defined on an NDArray > > object - so Scala users are not losing much by using null in place of None. > > You can create a implicit to maintain backward compatibility > > 2) It is adding 220+ APIs(I understand it is generated) for NDArray alone > > 3) this is adding another 100s of APIs unnecessarily, we are starting with > > NDArray but we can't stop there, we will have to do this for Symbol, > > Executor, Iterators, etc., . > > 3) I don't want to be fixing bugs and maintaining code in 2 places. > > 4) I want the cryptic code(# scala macros) to a minimum. > > 5) increased compilation time & bad developer experience - the time to > > compile has gone up quite a bit since we added the APIs last release on my > > 3 year old laptop already.. I think adding 400+ APIs unnecessarily would > > significantly increase build time and bad developer experience > > 6) I want to keep the core of the framework to be in Scala - because it > > allows you to write concise code - Yes it has a bit of learning curve, not > > everyone needs to know. I would rather invest in solidifying the Scala APIs > > and add more fe
Re: Time out for Travis CI
Kellen, I think this PR introduces extra runtime in CI, thus causes the timeout. Which means, once merged, every PR later will see same timeout in travis. So shall we modify the changes to decrease the test running time? or just disable the Travis CI? On Fri, Sep 28, 2018 at 9:17 PM Qin, Zhennan wrote: > > Hi Kellen, > > Thanks for your explanation. Do you have a time plan to solve the timeout > issue? Rebasing can't work for my case. Or shall we run it silently to > disallow it voting X for overall CI result? Because most developers are used > to ignore the PRs with 'X'. > > Thanks, > Zhennan > > -Original Message- > From: kellen sunderland [mailto:kellen.sunderl...@gmail.com] > Sent: Friday, September 28, 2018 10:38 PM > To: dev@mxnet.incubator.apache.org > Subject: Re: Time out for Travis CI > > Hey Zhennan, you're safe to ignore Travis failures for now. They're just > informational. > > The reason you sometimes see quick builds and sometimes see slow builds is > that we're making use of ccache in between builds. If your PR is similar to > what's in master you should build very quickly, if not it's going to take a > while and likely time out. If you see timeouts rebasing may speed things up. > Unfortunately the timeouts are global and we're not able to increase them. > I'm hoping that adding artifact caching will speed up future builds to the > point that test runs and builds can be executed in under the global limit > (which is ~50 minutes). > > -Kellen > > > On Fri, Sep 28, 2018 at 4:05 PM Qin, Zhennan wrote: > > > Hi MXNet devs, > > > > I'm struggled with new Travis CI for a while, it always run time out > > for this PR: > > https://github.com/apache/incubator-mxnet/pull/12530 > > > > Most of the time, Jenkins CI can pass, while Travis can't be finished > > within 50 minutes. For this PR, it shouldn't affect much on the build > > time or unit test time. Also, I saw other PR has same problem, eg. > > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/433172088?utm_sour > > ce=github_status&utm_medium=notification > > > > https://travis-ci.org/apache/incubator-mxnet/builds/434404305?utm_sour > > ce=github_status&utm_medium=notification > > > > According to the time stamp from Travis, all passed PR are within > > small code change, and can complete `make -j2` within 25s. But for > > timeout case, 'make -j2' will need about 1600s. Does Travis do > > incremental build for each test? Shall we increase time limit for > > large PR? Can we add more time stamp for build and unites stage to help > > understand what's going on there? > > > > Thanks in advance, > > Zhennan > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Feedback request for new Java API
lar to write these APIs, the discussion is > about using Scala Macros to generate 2 different types of APIs which are > functionally not different and usability wise are very very similar, look > at the example. > Thanks for your input, I will deposit your 0.02$ in our JIRA bank :) > > @Carin: It requires more effort to use AspectJ or similar to generate APIs > using reflection or at compile time, here we need to generate at compile > time so Java users have the API signature on their IDEs. > > Thanks, Naveen > > P.S: I am traveling and my responses will be delayed. > > > On Fri, Sep 28, 2018 at 10:25 AM Carin Meier wrote: > > > Sorry bad paste on the gist - here is the good one > > https://gist.github.com/gigasquid/01cd48f563db4739910592dd9ac9db20 > > > > On Fri, Sep 28, 2018 at 10:24 AM Carin Meier wrote: > > > > > +1 on option #2 > > > > > > In the case of minimizing the the overhead for code maintenance, I wanted > > > to suggest the option of investigating generating code from the Java > > > Reflection for the Java APIs. I did a quick gist from Clojure of what > > the > > > generated classes look like from the current Scala Symbol.api for > > > FullyConnected here > > > https://gist.github.com/gigasquid/01cd48f563db4739910592 > > > > > > I looks like that there is always a base Java class generated will all > > the > > > arguments. If this is the case, then there is a possibility to generate a > > > Java api based on this Java method automatically with just a conversion > > for > > > the Scala option and it might be reusable for all the packages. > > > > > > Not sure if it will work for this use case, but thought I would bring it > > > up in case it's helpful. > > > > > > - Carin > > > > > > On Fri, Sep 28, 2018 at 7:05 AM Davydenko, Denis > > > > > wrote: > > > > > >> +1 on option #2. Having clear Java interface for NDArray, from my > > >> perspective, would be a better experience for Java users as it won't > > >> require them to deal with Scala code in any capacity. Overhead of extra > > >> code for additional macros is justified, in my mind, as it will be > > >> introduced with option #1 either way, just in a different place. > > >> > > >> -- > > >> Thanks, > > >> Denis > > >> > > >> On 9/27/18, 6:14 PM, "YiZhi Liu" wrote: > > >> > > >> I vote for "2.) Leave the existing macro in place and add another > > >> which generates a Java friendly version" > > >> > > >> @Qing @Andrew, could you give some examples, so that people can > > better > > >> understand how it provides "best possible experience" to Java users. > > >> > > >> I have no strong preference between having JavaShape & JavaContext > > or > > >> not. > > >> On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres < > > >> andrew.f.ay...@gmail.com> wrote: > > >> > > > >> > That's not really the conversation I'm wanting to have. I want a > > >> discussion > > >> > about the macros with respect to NDArray so that we can get > > >> agreement on > > >> > our path forward with respect to implementing the NDArray wrapper. > > >> > > > >> > The design that was put forth and agreed to was for a a Java > > >> wrapper around > > >> > the Scala API. Adding a bunch of Java friendly methods inside the > > >> Scala > > >> > code would create a mess for users. Maintenance would be > > >> essentially the > > >> > same for both because either way you're going to be updating Java > > >> methods > > >> > when you make Scala changes. > > >> > > > >> > Let's please stick with the issue in the original email. > > >> > > > >> > Thanks, > > >> > Andrew > > >> > > > >> > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan > > >> wrote: > > >> > > > >> > > I would like to loop this back a layer. Current, there is a > > >> discussion in > > >> > > the MXNet Scala community on the ways to implement the Java > > APIs. > > >> Cu
Re: Feedback request for new Java API
I vote for "2.) Leave the existing macro in place and add another which generates a Java friendly version" @Qing @Andrew, could you give some examples, so that people can better understand how it provides "best possible experience" to Java users. I have no strong preference between having JavaShape & JavaContext or not. On Thu, Sep 27, 2018 at 5:56 PM Andrew Ayres wrote: > > That's not really the conversation I'm wanting to have. I want a discussion > about the macros with respect to NDArray so that we can get agreement on > our path forward with respect to implementing the NDArray wrapper. > > The design that was put forth and agreed to was for a a Java wrapper around > the Scala API. Adding a bunch of Java friendly methods inside the Scala > code would create a mess for users. Maintenance would be essentially the > same for both because either way you're going to be updating Java methods > when you make Scala changes. > > Let's please stick with the issue in the original email. > > Thanks, > Andrew > > On Thu, Sep 27, 2018 at 5:22 PM Qing Lan wrote: > > > I would like to loop this back a layer. Current, there is a discussion in > > the MXNet Scala community on the ways to implement the Java APIs. Currently > > there are two thoughts: > > > > 1. Make Scala Java Friendly (Create Java compatible methods in the Scala > > Class. such as NDArray with Java compatible constructor) > > 2. Make Java friendly wrappers in Scala (Andrew's explanation below) > > > > The first approach require minimum input from our side to implement > > however bring user a bunch of useless api they may not want to use. It also > > makes Scala package heavier. The good thing is these two packages require > > minimum maintenance cost. As a tradeoff, if any time in the future we want > > to make Java big (make Java as the primary language supported by MXNet), > > then the migration from Scala to Java will be harmful. Spark consider this > > carefully and decide not to change much on their Scala code base to make it > > more Java. > > > > The second approach will make unique NDArray, Shape, Context and more. The > > good thing about this is we can always holds a version control on Java. > > Some breaking changes on Scala may not influence much on Java. It did the > > best way to decouple the module and good for us to build unique pipeline > > for Java. The bad thing with this design is the maintenance cost as we need > > to keep two code bases, but it also make Java side easy to change to make > > it better compatible with users. > > > > Thanks, > > Qing > > > > On 9/27/18, 3:25 PM, "Andrew Ayres" wrote: > > > > Hi, > > > > Currently, we're working to implement a new Java API and would like > > some > > feedback from the community on an implementation detail. In short, the > > new > > Java API will use the existing Scala API (in a manner similar to how > > the > > current Clojure API works). This basically means that we're making Java > > friendly wrappers to call the existing Scala API. > > > > The feedback we're looking for is on the implementation of NDArray. > > Scala's > > NDArray has a significant amount of code which is generated via macros > > and > > we've got two viable paths to move forward: > > > > 1.) Change the macro to generate Java friendly methods - To do this > > we'll > > modify the macro so that the generated methods won't have > > default/optional > > arguments. There may also have to be some changes to parameter types to > > make them Java friendly. The big advantage here is that ongoing > > maintenance > > will easier. The disadvantages are that we'll be changing the existing > > Scala NDArray Infer API (it's marked experimental) and Scala users will > > lose the ability to use the default and optional arguments. > > > > 2.) Leave the existing macro in place and add another which generates a > > Java friendly version - The biggest issue here is that we'll be > > doubling > > the number of macros that we've got to maintain. It'll become even more > > overhead once we start expanding the Java API with more classes that > > use > > generated code like this. The advantages are that the existing Scala > > NDArray Infer API would remain unchanged for Scala users and that the > > new > > macro could be optimized to give the best possible experience to the > > Java > > API. > > > > Thanks, > > Andrew > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: [VOTE] Subscribe dev@ to Github Activities
+1 On Tue, Jul 17, 2018 at 12:55 PM Joshua Z. Zhang wrote: > > +1 > > We NEED to bring valuable discussions to a centralized place (@dev for > example) rather than scattered single threads. > Per filter options, there are a lot we can do to improve the SNR. > > Zhi > > On 2018/07/17 16:26:01, Sheng Zha wrote: > > Hi Anirudh, > > > > 1. You need exactly one filter to filter out all the github notifications > > on PRs and issues: "from:notificati...@github.com", and you'd get your S/N > > ratio back. > > 2. Having the option to do design discussion on an issue or PR is actually > > a good thing as many discussions are quite small and better accompanied by > > code. If for some reason a merged design needs revisiting, there's still > > the option of sending an email to dev@ and discuss about it. > > 3. About votes, commit vote (and veto) can already happen on PR per past > > agreement. The discussion for procedural vote IMO should be allowed to > > happen on Github if it's development related. Procedural votes themselves > > should and can still happen on dev@. > > > > About "you don't really have to do anything explicitly on the dev@ list", > > besides the above arguments, we don't send emails to dev@ just for the > > purpose of sending it. On the other hand, since "whatever didn't happen on > > dev list didn't happen", we'd need better arguments on why we'd choose to > > forego the transparency. > > > > -sz > > > > On Tue, Jul 17, 2018 at 8:47 AM, Anirudh wrote: > > > > > -1 > > > > > > The low signal to noise ratio would mean that we may miss important > > > emails. > > > Even with the different filters that we may setup for dev@, the emails > > > would be too many to not miss the important ones. We would see more and > > > more people starting a design discussion on an issue or PR. Because of the > > > low signal to noise ratio on the dev@ list, many may miss these > > > discussions. > > > > > > Slowly, this would erode the purpose of the dev@ list as this means that > > > you don't really have to do anything explicitly on the dev@ list. > > > You can start a design discussion on a github issue. You can start a > > > vote/discussion on a github issue. > > > > > > Anirudh > > > > > > On Mon, Jul 16, 2018 at 4:35 AM, Timur Shenkao wrote: > > > > > > > +1 if my vote can be taken into account > > > > > > > > On Mon, Jul 16, 2018 at 4:32 AM, Sheng Zha wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm starting a vote on subscribing dev@ to Github activities. See > > > > previous > > > > > discussion thread here > > > > > <https://lists.apache.org/thread.html/3d883f6a3cbc8e81e810962e0c0fe7 > > > > > bfd01f0b78d3cb44034f566442@%3Cdev.mxnet.apache.org%3E> > > > > > . > > > > > > > > > > The vote lasts for three days and ends on 7/18/2018 at 9pm pst. > > > > > > > > > > -sz > > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Proposal to release the Examples for Scala in V1.3
Agree to make incremental improvements. As long as the changes do not change the current states of memory leak, but improve the documents and demonstrate the way to use type-safe apis, I think it is fair to merge into v1.3. On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy wrote: > > Qing, its great that you are working on improving the Scala Examples with > documentation and tests. > Ideally, we shouldn't have any memory leaks..it erodes trust with our > users, however I understand it can take significant time and debugging > effort to resolve them. Given that these leaks already exist, may be should > call out on the documentation that this is a known issue and we will work > on it. > IMO for this case making incremental improvements to examples makes sense. > > I would like to hear what others think. > -Naveen > > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan wrote: > > > Hi all, > > > > I would like to propose the release plan for MXNet Scala examples in V1.3. > > Currently, the examples in the main repository are not maintained for a > > long time. I have spent some time to use the improved Scala API (known as > > the .api) to improve the examples with documentation to run and add CI > > tests. However, I am facing the difficulties to resolve memory leaks as > > most of the examples did not contains memory leak fixes. It crashes the CI > > because of that. We can temporarily disable the CI test for some of the > > examples and release them with 1.3 and enable them once we have a better GC > > in MXNet Scala. Here is a brief Pros and Cons for this release: > > > > Pros: > > > > * A good set of demos on how to use improved Training API > > * Examples are fixed with full documentation to help to run them > > > > Cons: > > > > * Examples contains the memory leak issues may cause problems > > > > Please kindly share your ideas in this thread and I am really appreciate > > for your help. > > > > Thanks, > > Qing > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Merging Clojure PR
Thanks Carin for the great contribution. I just left several questions in the PR, would you mind to take a look? Sorry for the late review. On Thu, Jun 28, 2018 at 12:14 PM Hagay Lupesko wrote: > > Thanks for your contribution Carin! > Unfortunately i can't do a proper review for Clojure, but it's great to see > the contribution and see how it develops and grows... > > On Thu, Jun 28, 2018 at 11:11 AM Carin Meier wrote: > > > Thanks everyone for your feedback and efforts with the Clojure package PR. > > > > I'm delighted to join the MXNet community and work with you all and am > > excited to invite the Clojure community to grow with it :) > > > > Thanks, > > Carin > > > > On Thu, Jun 28, 2018 at 1:38 PM, Pedro Larroy < > > pedro.larroy.li...@gmail.com> > > wrote: > > > > > Yes, great work Carin! I even saw your book on Clojure autographed. > > > > > > Pedro. > > > > > > On Wed, Jun 27, 2018 at 7:24 PM Naveen Swamy wrote: > > > > > > > Hi All, > > > > > > > > Carin (https://github.com/gigasquid) has done a worked on a Clojure > > > MXNet > > > > package for the Clojure community, Thank you Carin. > > > > > > > > I would like to merge this PR#11205 for the upcoming release 1.3. I am > > > not > > > > a Clojure developer and plan to just do a preliminary review for > > > licenses, > > > > tests, etc., and merge the code. Myself and Pedro called on help from > > the > > > > Clojure community in our day job at Amazon and also Carin's also got > > some > > > > peers from the Clojure community to help with the review. > > > > > > > > If there is a committer who would like to do a complete review, I'll be > > > > happy to step back and let you do it otherwise this PR is going in by > > the > > > > end of week to make it ready for 1.3. > > > > > > > > https://github.com/apache/incubator-mxnet/pull/11205 > > > > > > > > Let me know > > > > > > > > Thanks, Naveen > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: The operator check for Scala Package
> > > > > Hi Qing, > > > > > > > > > > thank you for working on improving the compatibility of our > > APIs! > > > > > > > > > > Your linked proposal does not describe the mentioned > > FILEHASH. > > > Could you > > > > > elaborate a bit? Would this be a hash of the entire file, > > some hash > > > > created > > > > > based on the signature of the underlying C++ methods or > > maybe a > > > different > > > > > approach? > > > > > > > > > > Also, at which step would developers be notified of the > > change? I'd > > > > propose > > > > > that we make this check a nightly job to prevent it from > > blocking > > > a PR > > > > and > > > > > forcing contributors who are not familiar with Scala having > > to > > > make a > > > > > change to the Scala package. This would allow us to follow up > > > > > asynchronously but still provide actionable events that one > > can be > > > > > subscribed to. > > > > > > > > > > Best regards, > > > > > Marco > > > > > > > > > > On Tue, Jun 19, 2018 at 11:00 AM Qing Lan < > > lanking...@live.com> > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I am one of the maintainer for MXNet Scala package. > > Currently I > > > am > > > > > > building up a hash-check system on the generated API > > through C. > > > The PR > > > > > is > > > > > > in this URL: > > > > > > https://github.com/apache/incubator-mxnet/pull/11239 > > > > > > A file named FILEHASH will be added to the Scala that > > created > > > the MD5 > > > > > > string of the generated API document. It will check the > > > signature of > > > > all > > > > > > the operators during Scala CI testing. The reason I am > > doing > > > this is to > > > > > > make sure Scala developers will always be reminded if > > there is an > > > > > operator > > > > > > name/argument changes happened in different PRs. More > > detailed > > > info > > > > > > explained in here: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/ > > > > > MXNet+Scala+API+Usability+Improvement > > > > > > > > > > > > Pros: > > > > > > This method will always help us keep backwards > > compatibility of > > > > operators > > > > > > for Scala > > > > > > Cons: > > > > > > Require update on the FILEHASH with new contents when > > there is an > > > > > operator > > > > > > change. Developers who changed the operator should `make > > > scalapkg` to > > > > > > update that file. > > > > > > > > > > > > Please leave any thoughts you may have for this design and > > feel > > > free to > > > > > > review the code. > > > > > > > > > > > > Thanks, > > > > > > Qing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: users@mxnet
t purposes. Some discussions will jump from users@ to > > > dev@, > > > > >>> but > > > > >>>> at a different level. So I wouldn't forward one mailing list to > > the > > > > >>> other. > > > > >>>> > > > > >>>> On Fri, Jun 15, 2018, 21:01 Marco de Abreu > > > > >>>> wrote: > > > > >>>> > > > > >>>>> I think nobody was opposed to it in the past, right? > > > > >>>>> > > > > >>>>> I'd propose that all emails automatically get copied to dev@ to > > > > >> ensure > > > > >>>>> high > > > > >>>>> visibility initially. What do you think? > > > > >>>>> > > > > >>>>> Sebastian schrieb am Fr., 15. Juni 2018, 20:51: > > > > >>>>> > > > > >>>>>> I have already proposed this many times in the past and would > > > > >>> strongly > > > > >>>>>> encourage it. > > > > >>>>>> > > > > >>>>>> -s > > > > >>>>>> > > > > >>>>>> On 15.06.2018 21:56, Sergio Fernández wrote: > > > > >>>>>>> Hi, > > > > >>>>>>> > > > > >>>>>>> is there any good reason why the podling doesn't have a users@ > > > > >>>> mailing > > > > >>>>>> list > > > > >>>>>>> yet? > > > > >>>>>>> > > > > >>>>>>> Honestly speaking, I'm not a big fan of the other tools the > > > > >> podling > > > > >>>> is > > > > >>>>>>> using. Slack and Web forums a cool tools, and I used them a lot > > > > >> in > > > > >>>>> other > > > > >>>>>>> contexts. But when it comes to transparency and community, > > > > >> mailing > > > > >>>>> lists > > > > >>>>>>> play a crucial role in the Apache Way. > > > > >>>>>>> > > > > >>>>>>> Users are the most important asset a project can have. Even > > more > > > > >>> than > > > > >>>>>>> developers, believe me. So I think it's time to create a users@ > > > > >>>>> mailing > > > > >>>>>>> list for to helping MXNet grow its community beyong the core > > > > >> team. > > > > >>>>>>> > > > > >>>>>>> Cheers, > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Regarding 1.2.1 patch release
xes that was made to publish > > to Maven > > > > > into > > > > > > 1.2.1 > > > > > > > -- currently in PR https://github.com/apache/ > > > > > > incubator-mxnet/pull/11147/ > > > > > > > > > > > > > > Also additionally, I would like to get the fix for > > > > > > > https://github.com/apache/incubator-mxnet/issues/10436 - > > > > currently > > > > > > another > > > > > > > contributor Andrew is working on it. > > > > > > > > > > > > > > -Naveen > > > > > > > > > > > > > > On Thu, Jun 7, 2018 at 8:33 AM, Lv, Tao A < > > tao.a...@intel.com> > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for bringing this up, Da! > > > > > > > > > > > > > > > > It would be great if we can have these fixes into > > 1.2.1 patch > > > > > > release, > > > > > > > > especially for https://github.com/apache/ > > > > > > incubator-mxnet/pull/10651, it > > > > > > > > has fixed > > > > https://github.com/apache/incubator-mxnet/issues/11028 > > > > > . > > > > > > > > > > > > > > > > What I want to add are: > > > > > > > > 1. https://github.com/apache/ > > incubator-mxnet/pull/10810 > > > which > > > > > > fixed > > > > > > > > https://github.com/apache/incubator-mxnet/issues/10809 > > . > > > > > > > > 2. https://github.com/apache/ > > incubator-mxnet/pull/10613 for > > > > doc > > > > > > > > improvement. > > > > > > > > > > > > > > > > I don't quite understand the criteria for choosing > > patch PRs > > > > and > > > > > > what's > > > > > > > > the timeline of 1.2.1 patch release. If it's possible, > > I also > > > > > want > > > > > > to > > > > > > > have > > > > > > > > below PRs. But it depends the progress of review and > > merging. > > > > > > > > 1. bug fix: > > > > https://github.com/apache/incubator-mxnet/pull/11095 > > > > > > (under > > > > > > > > review) > > > > > > > > 2. perf improvement: https://github.com/apache/ > > > > > > > incubator-mxnet/pull/11047 > > > > > > > > (approved) > > > > > > > > 3. doc improvement: https://github.com/apache/ > > > > > > incubator-mxnet/pull/11049 > > > > > > > > (under review) > > > > > > > > > > > > > > > > BR, > > > > > > > > -tao > > > > > > > > > > > > > > > > -Original Message- > > > > > > > > From: Zheng, Da [mailto:dzz...@amazon.com] > > > > > > > > Sent: Thursday, June 7, 2018 8:55 AM > > > > > > > > To: dev@mxnet.incubator.apache.org > > > > > > > > Cc: Emani, Ashok ; Lv, Tao A < > > > > > > tao.a...@intel.com > > > > > > > >; > > > > > > > > Zhao, Patric ; Zai, Alexander < > > > > > > alex...@amazon.com > > > > > > > > > > > > > > > > Subject: Re: Regarding 1.2.1 patch release > > > > > > > > > > > > > > > > Hello Anirudh, > > > > > > > > > > > > > > > > Could you merge the bugs below? Each of the bug fixes > > below > > > > come > > > > > > with a > > > > > > > > set of tests and many of them are critical for Gluon. > > > > > > > > MKLDNN: > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10979 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10731 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10706 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10651 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10624 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10619 > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10616 > > > > > > > > > > > > > > > > Others: > > > > > > > > https://github.com/apache/incubator-mxnet/pull/10918 > > > > > > > > > > > > > > > > @Ashok @Tao @Patric @Alex, do you have other important > > MKLDNN > > > > bug > > > > > > fixes > > > > > > > > that should be merged? > > > > > > > > > > > > > > > > Best, > > > > > > > > Da > > > > > > > > > > > > > > > > On 6/6/18, 5:23 PM, "Anirudh" > > > > > > > > anirudh2...@gmail.com>> wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I wanted to bring up some MKLDNN fixes that went > > into > > > > master > > > > > > but not > > > > > > > > into > > > > > > > > 1.2. > > > > > > > > Should these changes be going to the patch release > > ? We > > > had > > > > > > kept some > > > > > > > > changes from going into 1.2 branch since we were > > waiting > > > > for > > > > > > the test > > > > > > > > suite > > > > > > > > for MKLDNN. > > > > > > > > Is this test suite planned for the next major or > > minor > > > > > > release? Also, > > > > > > > > are > > > > > > > > there any critical MKLDNN bug fixes which are in > > master > > > and > > > > > > well > > > > > > > > tested and > > > > > > > > can go into the patch release ? > > > > > > > > > > > > > > > > Anirudh > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 6, 2018 at 1:48 PM, Anirudh < > > > > > anirudh2...@gmail.com > > > > > > > > > > > > > > anirudh2...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > As you may be aware, 1.2 has an undocumented > > backwards > > > > > > incompatible > > > > > > > > change > > > > > > > > > relating to saving and loading params. Please > > see: > > > > > > > > https://github.com/ > > > > > > > > > apache/incubator-mxnet/issues/11091. > > > > > > > > > > > > > > > > > > More details about the fix will be tracked here: > > > > > > > > https://issues.apache. > > > > > > > > > org/jira/browse/MXNET-518 > > > > > > > > > > > > > > > > > > The above fix will go as part of the 1.2.1 patch > > > release > > > > > > which will > > > > > > > > be > > > > > > > > > coming out soon. > > > > > > > > > > > > > > > > > > In addition to this, we will also be including: > > > > > > > https://github.com/ > > > > > > > > > apache/incubator-mxnet/pull/11142 > > > > > > > > > > > > > > > > > > I request the community to point out other bug > > fixes > > > that > > > > > are > > > > > > > > critical and > > > > > > > > > should go as part of 1.2.1 patch release. > > > > > > > > > > > > > > > > > > Anirudh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Vote to stop using JIRA
I feel so far our 'jira process' is not well organized. Though I agree with Steffen that Jira may be valuable for management task in some cases, it is not good to force every PR tie with a JIRA task. For example, I don't think this one worth a JIRA: https://github.com/apache/incubator-mxnet/pull/11225 . I did see PRs with only tens of line changes been blocked by reviewers, just because there's no related JIRA task. That's too brute and definitely not friendly to new contributors. If one (or a team) has a task need to be tracked, e.g., it is a large feature, or it will be improved or fixed later, etc, I think it might be fine to tracked by JIRA. If someone met problems, spent 10 minutes fixed, shot a PR, it does not make sense to stick a red-cross and say, a JIRA is required. On Mon, Jun 11, 2018 at 9:52 AM Timur Shenkao wrote: > > > I like GitHub for it's good community integration - just mention > somebody or link it in another issue/PR and everything will automatically > be referenced; pretty neat. > Take a look at > https://issues.apache.org/jira/projects/SPARK/issues/SPARK-14220?filter=allopenissues > > BTW Spark is also "complex" multi-language project. > They also had problems with long running builds and test jobs > > On Mon, Jun 11, 2018 at 5:23 PM, Marco de Abreu < > marco.g.ab...@googlemail.com> wrote: > > > Thanks for this detailed description! > > > > I think GitHub and JIRA are a good thing to co-exist. GitHub for issues and > > JIRA for action items (as a result of the issues). I like GitHub for it's > > good community integration - just mention somebody or link it in another > > issue/PR and everything will automatically be referenced; pretty neat. > > > > How about you set up a personal instance of JIRA (let us know if the public > > ones differ too much from the one we're running under Apache) and play > > around a bit? We can then review the setup and migrate the necessary > > changes after everything looks good. But sure, if anybody at @dev has > > further experience, I'd love to see them involved. > > > > -Marco > > > > On Mon, Jun 11, 2018 at 7:22 AM Yasser Zamani > > wrote: > > > > > > > > > > > On 6/11/2018 6:32 PM, Sebastian wrote: > > > > Looks like they also don't handle the committer nomination process > > > > correctly ... > > > > > > > > "If you are interested in becoming a Weex committer, contact any > > > > existing committer and we will help you go through the invitation > > > process." > > > > > > Yes you're absolutely right. Contributors wouldn't ask for this, > > > instead, they will be nominated. In some cases like this [1] we're > > > better than Weex :) > > > > > > [1] https://cwiki.apache.org/confluence/display/MXNET/ > > Becoming+a+Committer > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Clojure Package
gt; > >>> This is great effort and very nicely documented. When I >>>> > surveyed >>>> > > at >>>> > > > > my >>>> > > > > > >> day >>>> > > > > > >>> job on the interest for Scala packages there were a few >>>> people >>>> > > who >>>> > > > > > >>> specifically were interested in the Clojure packages, I >>>> think >>>> > > those >>>> > > > > are >>>> > > > > > >>> people who might have tried and understand the complexity >>>> of >>>> > > making >>>> > > > > > Scala >>>> > > > > > >>> work in Clojure. >>>> > > > > > >>> >>>> > > > > > >>> I am in support of a MXNet-Clojure package, I request you >>>> to >>>> > add >>>> > > > more >>>> > > > > > >> unit >>>> > > > > > >>> tests and integration tests that can be ported to CI. >>>> > > > > > >>> >>>> > > > > > >>> Thanks for your efforts. >>>> > > > > > >>> >>>> > > > > > >>> -Naveen >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> On Fri, Jun 1, 2018 at 6:16 PM, Carin Meier < >>>> > > carinme...@gmail.com> >>>> > > > > > >> wrote: >>>> > > > > > >>> >>>> > > > > > >>>> It is always a good thing to consider the cost with the >>>> > benefit. >>>> > > > > I'll >>>> > > > > > >> do >>>> > > > > > >>> my >>>> > > > > > >>>> best to explain what I see the tradeoffs to be. >>>> > > > > > >>>> >>>> > > > > > >>>> First, I wanted to clarify that it took significant >>>> > development >>>> > > > > effort >>>> > > > > > >> to >>>> > > > > > >>>> get the Clojure package and the interop working properly >>>> > despite >>>> > > > my >>>> > > > > > >>> simple >>>> > > > > > >>>> looking design on the confluence page :) >>>> > > > > > >>>> >>>> > > > > > >>>> One of the advantages of MXNet over its competitors is its >>>> > many >>>> > > > > > >> language >>>> > > > > > >>>> support. The Clojure package would only increase the >>>> value of >>>> > > this >>>> > > > > > >>>> proposition and bring new users and growth into the >>>> community. >>>> > > > > > >>>> However, there is a cost associated with adding this >>>> language >>>> > > > > support >>>> > > > > > >> as >>>> > > > > > >>>> you pointed out. >>>> > > > > > >>>> >>>> > > > > > >>>> Since the Clojure package right now is only reliant on the >>>> > Scala >>>> > > > > jars >>>> > > > > > >>> from >>>> > > > > > >>>> Maven, it can exist outside the main project as an >>>> independent >>>> > > > repo >>>> > > > > > >> but I >>>> > > > > > >>>> think that would lessen the growth benefit both to the >>>> Clojure >>>> > > > > > >> community >>>> > > > > > >>>> and to the MXNet community to not be included as a first >>>> class >>>> > > > > > >> language. >>>> > > > > > >>>> >>>> > > > > > >>>> I believe having first class Clojure support in MXNet is >>>> > > valuable, >>>> > > > > but >>>> > > > > > >>> the >>>> > > > > > >>>> cost of that support is up to the community to decide. >>>> > > > > > >>>> >>>> > > > > > >>>> Is there a process for considering a new package in MXNet? >>>> > > > > > >>>> >>>> > > > > > >>>> - Carin >>>> > > > > > >>>> >>>> > > > > > >>>>> On Fri, Jun 1, 2018 at 5:51 PM, Chen HY < >>>> > chenhy12...@gmail.com >>>> > > > >>>> > > > > > wrote: >>>> > > > > > >>>>> >>>> > > > > > >>>>> Have checked the issue and the confluence page, but still >>>> > > > curious. >>>> > > > > > >>>>> Clojure and Scala are both JVM based languages. >>>> > > > > > >>>>> They, as well as many JVM based languages, can share >>>> their >>>> > > class >>>> > > > > and >>>> > > > > > >>>> method >>>> > > > > > >>>>> at a certain level. >>>> > > > > > >>>>> Why should the community maintain two APIs for two >>>> languages >>>> > > with >>>> > > > > can >>>> > > > > > >>>> share >>>> > > > > > >>>>> their packages with almost zero effort? >>>> > > > > > >>>>> >>>> > > > > > >>>>> >>>> > > > > > >>>>> 2018-06-01 21:58 GMT+01:00 Carin Meier < >>>> carinme...@gmail.com >>>> > >: >>>> > > > > > >>>>> >>>> > > > > > >>>>>> Hi all, >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> I've been working on a Clojure package for MXNet. Since >>>> > > Clojure >>>> > > > is >>>> > > > > > >> a >>>> > > > > > >>>> JVM >>>> > > > > > >>>>>> language, the package leverages the great work of the >>>> > existing >>>> > > > > > >> Scala >>>> > > > > > >>>>>> package. >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> I would appreciate any feedback and testing. >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> Here is the original issue: >>>> > > > > > >>>>>> https://github.com/apache/incubator-mxnet/issues/8971 >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> Architecture & Design: >>>> > > > > > >>>>>> https://cwiki.apache.org/confluence/display/MXNET/ >>>> > > MXNet+Clojure >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> and the github repo for rapid testing and issue fixing >>>> > before >>>> > > of >>>> > > > > > >>>> opening >>>> > > > > > >>>>> an >>>> > > > > > >>>>>> official PR https://github.com/gigasquid/clojure-mxnet >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> I'm also active in the slack channel so feel free to >>>> ping me >>>> > > > > there. >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> Thanks, >>>> > > > > > >>>>>> Carin Meier >>>> > > > > > >>>>>> >>>> > > > > > >>>>> >>>> > > > > > >>>>> >>>> > > > > > >>>>> >>>> > > > > > >>>>> -- >>>> > > > > > >>>>> Chen Hanyang 陈涵洋 >>>> > > > > > >>>>> Software School Fudan University >>>> > > > > > >>>>> +86-138-1881-7745 >>>> > > > > > >>>>> >>>> > > > > > >>>> >>>> > > > > > >>> >>>> > > > > > >> >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > -- >>>> > > > > > > Sandeep Krishnamurthy >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> >>>> >>>> >>>> -- >>>> Chen Hanyang 陈涵洋 >>>> Software School Fudan University >>>> +86-138-1881-7745 >>>> >>> >>> >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: The New Scala API
I personally like the design here. Since we have seen technical difficulties of compatibility, I would like to ask people pay attention to the 'How to combine with existing APIs' section: https://cwiki.apache.org/confluence/display/MXNET/MXNet+Scala+API+Usability+Improvement#MXNetScalaAPIUsabilityImprovement-HowtocombinewithexistingAPIs Qing proposed three options, 1. Add a new Class/Object called "NewSymbol/NDArray" with full implementation. 2. Create a new Class and change the name space for all of the functions (e.g Activation -> NewActivation) and let Symbol/NDArray extends that. 3. Create a new Class and override the Same functions with different implementations. If we have to choose from option 1 and 2, I would like to +0.5 for option 2, with which users can quickly aware of the new easy-to-use API: they type 'Symbol.' in IDE as usual and these functions pop up. 2018-04-19 10:58 GMT-07:00 Qing Lan : > Hi All, > > I am Qing, one of the Scala API maintainer for MXNet. I would like to propose > a new design on Scala APIs, it will be really helpful for user to use MXNet > Symbol/NDArray. This is a follow-up from Naveen’s proposal. > > Background: > The current design on Scala would take arguments as key-value pair and didn’t > provide the type information for different arguments. There are document > missing for different functions which makes it even hard to use. > > Our approach: > We will provide a better designed Scala API for user to use with full > documentation and arguments definition. All arguments will be specifically > targeted to different functions. Please see one example that we show in the > Wiki<https://cwiki.apache.org/confluence/display/MXNET/MXNet+Scala+API+Usability+Improvement> > and leave any thoughts you may have. This wiki includes examples, targets > and scenarios we have so far. > > Thanks, > Qing -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: [VOTE] Disconnect all non-C API's from mxnet versioning
Kellen, we are not talking about using wrong native package AFTER downloading the package. It's about deciding which version to use BEFORE downloading, and collecting information to debug. Copy-paste my previous words, " 1. It is harmful to user experience 1) Each time users want to use a specific feature, need to first check the mxnet core version, then check which frontend work with this core version. 2) Each time users have problem using a frontend (Scala/R/...) API, need to figure out which core version they are using. And as I describe in my #5. Imagine an inverse situation. When someone has a model trained by gluon 1.6.0, he want to deploy it to JVM, what Scala API version should he use? 1.6.0? No. And which R package version he should use? It is still different from either Gluon version or Scala API version. What a nightmare. " 2018-03-12 16:10 GMT-07:00 kellen sunderland : > @Rahul + Roshani, I would hear what you're saying if the user had to worry > about using the native package, but that worry is abstracted from them. > The scala package has a dependency on the native library and includes the > native lib inside the jar. The correct lib is then bound against at > runtime. I don't see how a user can use the wrong library or be confused > here unless the instructions on this page are incorrect: > https://github.com/apache/incubator-mxnet/tree/master/scala-package > > On Tue, Mar 13, 2018 at 12:02 AM, Rahul Huilgol > wrote: > >> -1 for the frontends having different versions than the backend. It not >> only creates confusion for new users, but also increases the work of >> developers who need to ensure compatibility. All this for a one-time change >> of namespace of a package? >> >> I think we should increase the major version number to make this change? >> Why do we have to 'wait' for 2.0? Who tells us that it's time for a 2.0 >> version? >> >> I think expecting a user to look up version numbers on the website and >> ensure compatibility as suggested above, is not a simple task. Most users >> might not even know how the backend and frontend integrate. They might not >> even know that there is a C API which powers the frontends. Even knowing >> how to look up documentation for a particular version of MXNet is a >> non-trivial task right now. (And there are pages in a version's >> documentation which link to a file in another version). We should avoid >> introducing more complexity into the process. As developers we tend to >> overlook the important aspect of user experience. I think we should take a >> step back and look at this from the perspective of a user, not from that of >> a developer who works closely with MXNet. >> >> Regards, >> Rahul >> >> On Mon, Mar 12, 2018 at 3:12 PM, Roshani Nagmote < >> roshaninagmo...@gmail.com> >> wrote: >> >> > -1 for different versioning. >> > >> > I feel its just added confusion for users. >> > >> > On Mon, Mar 12, 2018 at 2:35 PM, YiZhi Liu wrote: >> > >> > > Agree. >> > > >> > > And my reply to Marco's point, >> > > >> > > > Changing namespaces is one use-case, but there will be a lot more >> with >> > > increasing activity - we have to take the bigger picture in mind. >> > > And you mentioned the CPP package as an example. >> > > > During analysis, we figured that a re-engineering of that API would >> be >> > > more appropriate and easier maintainable. >> > > I cannot agree as an engineer. Why not keep old API and add new ones? >> > > Just like in c_api.h, we added xxxEx while did not remove xxx, which >> > > keeps APIs compatible. >> > > >> > > > I think it is safe to say that the other APIs have not been >> maintained >> > > as actively as our Python/Gluon API. >> > > Are you saying, if an API is maintained actively and is widely used, >> > > then it should be versioned together with MXNet Core? >> > > Interesting, maybe instead we should have another discussion to decide >> > > whether to remove some of the 'inactive' frontend bindings from the >> > > repo. >> > > >> > > > We have to do #3 anyways, so it is just about having a different >> number >> > > set as version string. >> > > A release with 6 different versions and 5 mappings? >> > > >> > > > I really don't see an issue in #1 - it's a simple lookup that could >> be >> > > done on our website.
Re: [VOTE] Disconnect all non-C API's from mxnet versioning
Agree. And my reply to Marco's point, > Changing namespaces is one use-case, but there will be a lot more with > increasing activity - we have to take the bigger picture in mind. And you mentioned the CPP package as an example. > During analysis, we figured that a re-engineering of that API would be more > appropriate and easier maintainable. I cannot agree as an engineer. Why not keep old API and add new ones? Just like in c_api.h, we added xxxEx while did not remove xxx, which keeps APIs compatible. > I think it is safe to say that the other APIs have not been maintained as > actively as our Python/Gluon API. Are you saying, if an API is maintained actively and is widely used, then it should be versioned together with MXNet Core? Interesting, maybe instead we should have another discussion to decide whether to remove some of the 'inactive' frontend bindings from the repo. > We have to do #3 anyways, so it is just about having a different number set > as version string. A release with 6 different versions and 5 mappings? > I really don't see an issue in #1 - it's a simple lookup that could be done > on our website. Please be careful to say 'simple', each time we introduce an additional step, we lose a number of our potential users. And as I describe in my #5. Imagine an inverse situation. When someone has a model trained by gluon 1.6.0, he want to deploy it to JVM, what Scala API version should he use? 1.6.0? No. And which R package version he should use? It is still different from either Gluon version or Scala API version. What a nightmare. 2018-03-12 14:11 GMT-07:00 Chris Olivier : > Marco, you're mixing votes again. > > > > > > > > > > > * This leaves us with three options: 1. Vote failed: No refactoring of > user-facing APIs (including namespace changes) possible OR major version > increase 2. Vote succeeded: Refactoring of user-facing APIs possible and > only users of the changed APIs are affected while major version does not > increase for other APIs. 3. Remove SemVer: We could introduce breaking > changes at any point in time, but our users would be losing trust due to > unexpected failures during upgrades.* > > What you're describing is not what this vote is about. This vote is > whether to separate mxnet and API versioning. > Please try to stay on task. > > > > On Mon, Mar 12, 2018 at 1:56 PM, Marco de Abreu < > marco.g.ab...@googlemail.com> wrote: > >> Regarding #4: Changing namespaces is one use-case, but there will be a lot >> more with increasing activity - we have to take the bigger picture in mind. >> I think it is safe to say that the other APIs have not been maintained as >> actively as our Python/Gluon API (which I would say could be versioned >> together with MXNet Core, but it does not really make a difference). This >> results in our APIs not reflecting all features available in MXNet (#2) or >> doing it in a way that we wouldn't recommend nowadays. While it is no >> problem to add new features to an API using a minor version change, it >> limits our possibilites to do a refactor. Our team, for example, got a >> customer that would like to see the functionality of the Cpp package being >> increased. During analysis, we figured that a re-engineering of that API >> would be more appropriate and easier maintainable. If we don't pass this >> vote, we won't be able to make any improvements to our less maintained APIs >> without a major version increment - which the community is also heavily >> against. We have to do #3 anyways, so it is just about having a different >> number set as version string - right now we're making it easy for ourselves >> by basically not maintaining any other than the Python interface and >> declining all breaking changes or refactors to APIs. I really don't see an >> issue in #1 - it's a simple lookup that could be done on our website. >> Simply select the version of MXNet you would like to have and it will >> provide you with the appropriate installation instructions - the same way >> we're already doing it. >> >> This leaves us with three options: >> 1. Vote failed: No refactoring of user-facing APIs (including namespace >> changes) possible OR major version increase >> 2. Vote succeeded: Refactoring of user-facing APIs possible and only users >> of the changed APIs are affected while major version does not increase for >> other APIs. >> 3. Remove SemVer: We could introduce breaking changes at any point in time, >> but our users would be losing trust due to unexpected failures during >> upgrades. >> >> -Marco >> >> >> On Mon, Mar
Re: [VOTE] Disconnect all non-C API's from mxnet versioning
STRONGLY -1 (binding) as I explained in another thread 'Publishing Scala Package/namespace change'. I think separating version is harmful. 1. It is harmful to user experience 1) Each time users want to use a specific feature, need to first check the mxnet core version, then check which frontend work with this core version. 2) Each time users have problem using a frontend (Scala/R/...) API, need to figure out which core version they are using. 2. Frontend APIs are tightly binding to the 'MXNet Core', e.g., almost all APIs extract operator definitions from the Core, which makes the API version and Core version a one-on-one mapping. Then why separate? 3. It introduces overhead for release. Now each release need to involve a bunch of frontend release version, along with the MXNet core release version. 4. The only benefit I see so far is, it makes easier for Scala package to change the namespace from ml.dmlc to org.apache (by increasing Scala API major version id without changing the MXNet core major version). But, 1) It is very likely that, this is the ONLY time we benefit from separate versioning. Changing namespace is a very rare issue that forces us to make APIs incompatible. In other situations, the APIs evolves smoothly which can stay compatible for a long time. 2) We can still discuss whether we have to change the major version. 3) Even the answer to 2) is Yes, I think it is affordable to wait for MXNet 2.0 to change 'ml.dmlc' to 'org.apache' 5. Other Apache projects, e.g., Apache Spark, have PySpark (Python frontend API), SparkR (R frontend API), MLLib, GraphX, etc, same version as the Spark Core, as well as the Scala/Java API. I feel it convenient since every time I check a document, say, MLLib 1.6.0, I can tell it works with Spark Core 1.6.0 and GraphX 1.6.0. And I can expect when I use Python API 1.6.0, it will behave the same. and for +1 votings, do you mean to separate Python/Gluon API versioning as well? 2018-03-12 11:18 GMT-07:00 Naveen Swamy : > -1 for different versioning, it not only be maintenance nightmare but also > more importantly confusing to users, > > > On Mon, Mar 12, 2018 at 9:57 AM, Marco de Abreu < > marco.g.ab...@googlemail.com> wrote: > >> According to the discussion in the Scala thread, the release cycles would >> stay unchanged and are still part of the mxnet releases. >> >> Nan Zhu schrieb am Mo., 12. März 2018, 17:42: >> >> > how about release cycle? >> > >> > >> > On Mon, Mar 12, 2018 at 9:37 AM, Yuan Tang >> > wrote: >> > >> > > +1 >> > > >> > > On Mon, Mar 12, 2018 at 12:35 PM, Marco de Abreu < >> > > marco.g.ab...@googlemail.com> wrote: >> > > >> > > > +1 >> > > > >> > > > Tianqi Chen schrieb am Mo., 12. März >> 2018, >> > > > 17:33: >> > > > >> > > > > +1 >> > > > > >> > > > > On Mon, Mar 12, 2018 at 9:32 AM, Chris Olivier < >> > cjolivie...@apache.org >> > > > >> > > > > wrote: >> > > > > >> > > > > > It has been proposed that all Non-C API's follow separate >> > versioning >> > > > from >> > > > > > the main mxnet C API/releases. >> > > > > > >> > > > > > A +1 vote is in *favor of* using a different versioning for all >> > > > > > non-C-API's, with each API (Scala, R, Julia, C++, etc.) having >> its >> > > own >> > > > > > version. >> > > > > > >> > > > > > A -1 vote is *against* using a different versioning for all >> > > > non-C-API's, >> > > > > > with all API's (Scala, R, Julia, C++, etc.) sharing the mxnet >> > > version. >> > > > > > >> > > > > > This vote will conclude on Monday, March 19, 2018. >> > > > > > >> > > > > > Thanks, >> > > > > > -Chris >> > > > > > >> > > > > >> > > > >> > > >> > >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Publishing Scala Package/namespace change
And just to make clear why I raise Spark MLLib as an example. I was arguing that namespace changing is a very rare issue that we have to make APIs incompatible. For other situations, the APIs evolves smoothly, MLLib '@deprecated' 'train(RDD)', instead of removing it, and make APIs compatible for a long time. 2018-03-11 14:26 GMT-07:00 Chris Olivier : > Since someone mentioned ABI, keep in mind that API compatibility does not > necessarily mean ABI compatibility. libpython, for example, may, within a > major version, guarantee backwards API compatibility (you can still compile > successfully), but does not guarantee ABI compatibility, as structure > sizes may change, for example. > > On Sun, Mar 11, 2018 at 1:56 PM kellen sunderland < > kellen.sunderl...@gmail.com> wrote: > > > "Did ML lib increase their major version after deprecating RDD?" > > > > Answering my own question. They will increase major version after RDD is > > removed. This is basically scenario 1 from above. It would mean we > > release MXNet 2.0 with the Scala changes. > > > > On Sun, Mar 11, 2018 at 9:54 PM, kellen sunderland < > > kellen.sunderl...@gmail.com> wrote: > > > > > "Why refactoring and deprecating means separating version from mxnet > > > core? Apache Spark MLLib refactors and deprecates a lot (e.g., they > > > deprecates RDD API), our C API also deprecates things, remember there > > are a > > > bunch of xxxEx in c_api.h?" > > > > > > Did ML lib increase their major version after deprecating RDD? > > > > > > "They will. Scala API runs auto code-generation to extract Symbol > method > > > from MXNet core. For example, users can write and compile > > > Symbol.NewOperator with one Scala API version, but they cannot run it > > with > > > an mxnet-core .so which does not have NewOperator / or have NewOperator > > > with different args." > > > > > > Not sure I fully understand the scenario you're describing here. Is > this > > > the case where a user writes a new operator against one version of > > > libmxnet.so and then runs it on an older version? In this case they'd > > need > > > to set a dependency on the current libmxnet.so ABI that they're running > > > against, and ensure that their jar was using that version or newer. > This > > > is the goal of SemVer per interface. > > > > > > "By doing major version change to Scala API, we remind users 'hey, be > > > careful, we have something incompatible!' But then what?" > > > They either choose to update their package and then fix potential > > breaking > > > API changes (the likely case), or they stick with the current version. > > > > > > "Users get more confused with the version mapping. And it introduces > > > overhead to maintain." > > > I'm not sure why users even need to know about the version mapping. If > > > I'm only interested in the Scala package from maven, why do I care > which > > > version of libmxnet.so I'm using? > > > > > > > > > > > > On Sun, Mar 11, 2018 at 8:06 PM, YiZhi Liu > wrote: > > > > > >> > > > >> > Changing namespaces is one example of a required major version > change, > > >> but > > >> > there are more reasons like general refactoring or some deprecated > > APIs > > >> > just being hard to maintain. > > >> > > >> Why refactoring and deprecating means separating version from mxnet > > core? > > >> Apache Spark MLLib refactors and deprecates a lot (e.g., they > deprecates > > >> RDD API), our C API also deprecates things, remember there are a bunch > > of > > >> xxxEx in c_api.h? > > >> > > >> They won't get a strange error, assuming we're talking about Scala > users > > >> > who are upgrading from a package with the same namespace they will > > rely > > >> on > > >> > the package manager to give them an update which should be painless. > > >> > > >> They will. Scala API runs auto code-generation to extract Symbol > method > > >> from MXNet core. For example, users can write and compile > > >> Symbol.NewOperator with one Scala API version, but they cannot run it > > with > > >> an mxnet-core .so which does not have NewOperator / or have > NewOperator > > >> with different a
Re: Publishing Scala Package/namespace change
> > I'm not sure why users even need to know about the version mapping. If I'm > only interested in the Scala package from maven, why do I care which > version of libmxnet.so I'm using? If users want to use a specific feature, e.g., Volta GPU support, they need to first check which MXNet core version brings the feature, then check which Scala API version works with this MXNet Core version. And vice versa, as Chris described, if they have problem using the Scala API, they need to figure out which core version they are using. They either choose to update their package and then fix potential breaking > API changes (the likely case), or they stick with the current version. Then what do we benefit from separating the version? They will increase major version after RDD is > removed. This is basically scenario 1 from above. It would mean we > release MXNet 2.0 with the Scala changes. > So, 1. Is it crucial enough to change major version for Scala API namespace changing? 2. If so, can we postpone changing the namespace and wait for MXNet 2.0 release? 2018-03-11 13:56 GMT-07:00 kellen sunderland : > "Did ML lib increase their major version after deprecating RDD?" > > Answering my own question. They will increase major version after RDD is > removed. This is basically scenario 1 from above. It would mean we > release MXNet 2.0 with the Scala changes. > > On Sun, Mar 11, 2018 at 9:54 PM, kellen sunderland < > kellen.sunderl...@gmail.com> wrote: > > > "Why refactoring and deprecating means separating version from mxnet > > core? Apache Spark MLLib refactors and deprecates a lot (e.g., they > > deprecates RDD API), our C API also deprecates things, remember there > are a > > bunch of xxxEx in c_api.h?" > > > > Did ML lib increase their major version after deprecating RDD? > > > > "They will. Scala API runs auto code-generation to extract Symbol method > > from MXNet core. For example, users can write and compile > > Symbol.NewOperator with one Scala API version, but they cannot run it > with > > an mxnet-core .so which does not have NewOperator / or have NewOperator > > with different args." > > > > Not sure I fully understand the scenario you're describing here. Is this > > the case where a user writes a new operator against one version of > > libmxnet.so and then runs it on an older version? In this case they'd > need > > to set a dependency on the current libmxnet.so ABI that they're running > > against, and ensure that their jar was using that version or newer. This > > is the goal of SemVer per interface. > > > > "By doing major version change to Scala API, we remind users 'hey, be > > careful, we have something incompatible!' But then what?" > > They either choose to update their package and then fix potential > breaking > > API changes (the likely case), or they stick with the current version. > > > > "Users get more confused with the version mapping. And it introduces > > overhead to maintain." > > I'm not sure why users even need to know about the version mapping. If > > I'm only interested in the Scala package from maven, why do I care which > > version of libmxnet.so I'm using? > > > > > > > > On Sun, Mar 11, 2018 at 8:06 PM, YiZhi Liu wrote: > > > >> > > >> > Changing namespaces is one example of a required major version change, > >> but > >> > there are more reasons like general refactoring or some deprecated > APIs > >> > just being hard to maintain. > >> > >> Why refactoring and deprecating means separating version from mxnet > core? > >> Apache Spark MLLib refactors and deprecates a lot (e.g., they deprecates > >> RDD API), our C API also deprecates things, remember there are a bunch > of > >> xxxEx in c_api.h? > >> > >> They won't get a strange error, assuming we're talking about Scala users > >> > who are upgrading from a package with the same namespace they will > rely > >> on > >> > the package manager to give them an update which should be painless. > >> > >> They will. Scala API runs auto code-generation to extract Symbol method > >> from MXNet core. For example, users can write and compile > >> Symbol.NewOperator with one Scala API version, but they cannot run it > with > >> an mxnet-core .so which does not have NewOperator / or have NewOperator > >> with different args. > >> > >> By doing major version change to Scala API, we remind users 'hey, be &g
Re: Publishing Scala Package/namespace change
> > > > My (non-binding) preference would be for option 2. > > > > > > -Kellen > > > > > > On Sun, Mar 11, 2018 at 12:44 PM, Marco de Abreu < > > > marco.g.ab...@googlemail.com> wrote: > > > > > >> Changing namespaces is one example of a required major version change, > > but > > >> there are more reasons like general refactoring or some deprecated > APIs > > >> just being hard to maintain. Things like these happen quite frequently > > and > > >> it's a problem every software project has to face and find a solution > > for. > > >> > > >> Regarding ' How to tell which Scala API version works with which MXNet > > >> core > > >> version?': We could just bundle MXNet with the released API package as > > we > > >> do right now, but we would give each interface it's own version and > > >> publish > > >> them on their distribution platforms accordingly. Just an example: > > >> >Scala-Package -> MXNet-Version > > >> >> 1.0 -> 1.0 > > >> >> 1.1 -> 1.1 > > >> >> 2.0 -> 1.2 > > >> >> 2.1 -> 1.3 > > >> >> 3.0 -> 2.0 > > >> > > >> > R-Package -> MXNet-Version > > >> >> 1.0 -> 1.0 > > >> >> 2.0 -> 1.1 > > >> >> 2.1 -> 1.2 > > >> >> 2.2 -> 1.3 > > >> >> 3.0 -> 2.0 > > >> > > >> This is always an N-to-1 mapping, while N being the versions of our > APIs > > >> and 1 the MXNet Core version. From MXNets versioning perspective, this > > >> would then looking the following: > > >> > MXNet-Version -> APIs > > >> >> 1.0 -> Scala_1.0; R_1.0 > > >> >> 1.1 -> Scala_1.1; R_2.0 > > >> >> 1.2 -> Scala_2.0; R_2.1 > > >> >> 1.3 -> Scala_2.1; R_2.2 > > >> >> 2.0 -> Scala_3.0; R_3.0 > > >> > > >> This would give us the liberty to develop MXNet without restricting us > > too > > >> much - of course, major version increments will still have to be > > >> considered > > >> carefully. I don't think that this would harm transparency too much > and > > >> there's no need to write big documentation. > > >> > > >> -Marco > > >> > > >> > > >> On Sun, Mar 11, 2018 at 12:16 PM, YiZhi Liu > > wrote: > > >> > > >> > I have no idea how separating Scala API version can solve the > > >> > 'compatibility' problem. Do you mean we have different version for > > >> > 'ml.dmlc' namespace and 'org.apache' namespace? Do these two > versions > > >> > have same behavior? How to tell which Scala API version works with > > >> > which MXNet core version? By document? How many users will read the > > >> > whole document and carefully pair the version id before they run > into > > >> > a strange error and give up? > > >> > > > >> > Moreover, changing namespace is an issue that is really rare and > > >> > hardly happens. For other 'compatibility' problem, for example, the > > >> > class/function definitions, should handle the compatibility itself. > > >> > You'll never expect a project to have a different version for > changing > > >> > 'calculate(int)' to 'calculate(float)', it should just add a new > > >> > function 'calculate(float)'. > > >> > > > >> > Regarding 'In this case the Scala interface is clearly a separate > > >> > entity from the C API.'. Everything can be seen as a separate > entity, > > >> > the mxnet engine, the graph description, operators, python API, > gluon > > >> > API, etc. We should think carefully what we want to provide, and > what > > >> > our users need. > > >> > > > >> > As an example, Apache Spark, still has SparkR (R API), PySpark > (Python > > >> > API), MLLib, GraphX ... as part of its release, and have the same > > >> > version as Spark core as well as its Scala/Java API. > > >> > > > >> > 2018-03-10 23:58 GMT-08:00 kellen sunderland < > > >> kellen.sunderl...@gmail.com > > >> > >: &g
Re: Publishing Scala Package/namespace change
I have no idea how separating Scala API version can solve the 'compatibility' problem. Do you mean we have different version for 'ml.dmlc' namespace and 'org.apache' namespace? Do these two versions have same behavior? How to tell which Scala API version works with which MXNet core version? By document? How many users will read the whole document and carefully pair the version id before they run into a strange error and give up? Moreover, changing namespace is an issue that is really rare and hardly happens. For other 'compatibility' problem, for example, the class/function definitions, should handle the compatibility itself. You'll never expect a project to have a different version for changing 'calculate(int)' to 'calculate(float)', it should just add a new function 'calculate(float)'. Regarding 'In this case the Scala interface is clearly a separate entity from the C API.'. Everything can be seen as a separate entity, the mxnet engine, the graph description, operators, python API, gluon API, etc. We should think carefully what we want to provide, and what our users need. As an example, Apache Spark, still has SparkR (R API), PySpark (Python API), MLLib, GraphX ... as part of its release, and have the same version as Spark core as well as its Scala/Java API. 2018-03-10 23:58 GMT-08:00 kellen sunderland : > +1 (non-binding) to what Marco is describing. +1 (non-binding) to getting > the Scala bindings with the namespace change into Maven. > > The general best practice for SemVer, which is used by most projects that > employ SemVer, is to apply SemVer to the public APIs of packages that ship > with your project. If you have several independent APIs this could mean that > they are versioned separately from each other, and from the overall project > versioning mechanism. > > For example, the .NET Core library ships with a number of binaries, each with > their own SemVerioned APIs. They also have a high-level, easy to understand > version for the package as a whole: > https://docs.microsoft.com/en-us/dotnet/core/versions/. > > Nodesource has a good description of this: > http://nodesource.com/blog/semver-a-primer/ > “Semver is a scheme for interface versioning for the benefit of interface > consumers, thus if a tool has multiple interfaces, e.g. an API and a CLI, > these interfaces may evolve independent versioning.” > > SemVer at its core is a communication mechanism to inform developers of > incompatibilities. In this case the Scala interface is clearly a separate > entity from the C API. I.e. changing the Scala namespace isn’t going to > break C API users. It does not communicate anything useful to these users if > we up their major version in response to a Scala change, it simply breaks > compatibility. If we group all interfaces together, and increment whenever > any of them has a breaking change we’ll soon be at MXNet version 587. We’ll > be forcing our users to check compatibility and update their dependency > tracking constantly. The end result is that our users will stop pulling in > new versions of the library. > > What I would propose is that (1) we have a high-level SemVer system that > tracks our C_API. This is the ‘MXNet’ version that we generally refer to and > emphasize for our public releases. For each API we have an independent > versioning system that if we can, we fix to the MXNet version. When it makes > sense we version these APIs independently. So for example we could have a > MXNet 1.2 release that ships with a 2.0 Scala API / R API. > > In terms of Apache process I think shipping artifacts with a non-Apache > namespace is a bigger issue than whatever versioning conventions we decide to > use. > > -Kellen > > From: Carin Meier > Sent: Saturday, March 10, 2018 1:41 PM > To: dev@mxnet.incubator.apache.org > Cc: d...@mxnet.apache.org > Subject: Re: Publishing Scala Package/namespace change > > +1 as well. I'm actively developing a Clojure package for MXNet that uses > the jars from the Scala package. > > - Carin > > On Fri, Mar 9, 2018 at 4:44 PM, YiZhi Liu wrote: > >> +1 for changing the namespace asap. for the maven deploy, we can have >> it build along with pip deployment. >> >> >> 2018-03-09 10:15 GMT-08:00 Naveen Swamy : >> > Hi Guys, >> > >> > I am working on MXNet Scala Inference APIs >> > <https://issues.apache.org/jira/browse/MXNET-50> along with another >> > contributor Roshani. A while back I noticed that we haven't been >> publishing >> > the scala package to Maven for a while now(last one being v0.11.1a under >> > the dmlc namespace). >> > Currently users hav
Re: Publishing Scala Package/namespace change
+1 for changing the namespace asap. for the maven deploy, we can have it build along with pip deployment. 2018-03-09 10:15 GMT-08:00 Naveen Swamy : > Hi Guys, > > I am working on MXNet Scala Inference APIs > <https://issues.apache.org/jira/browse/MXNET-50> along with another > contributor Roshani. A while back I noticed that we haven't been publishing > the scala package to Maven for a while now(last one being v0.11.1a under > the dmlc namespace). > Currently users have to build the package manually and then use it, this > hinders adoption and also is painful to build everything from source. > > I also see that we haven't changed the namespace to org.apache and instead > are still ml.dmlc namespace. > > I wanted to seek your opinion about changing the MXNet-Scala package > namespace to org.apache for the Scala package and publish to Maven in the > upcoming release. I understand that this probably breaks the Semver > semantics that is agreed upon, However I would like to point out that the > Scala package has never been published to maven as 1.0 under org.apache. > > Open to suggestions. > > Thanks, Naveen -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Assign JIRA issue
Yes it is. Thanks! 2018-03-08 16:04 GMT-08:00 Chris Olivier : > Done. Assume you meant your apache.org onew (there there two) > > On Thu, Mar 8, 2018 at 3:47 PM, YiZhi Liu wrote: > >> Hi Chris, >> >> could you help to add me to the permission group (committer)? >> >> Thanks, >> Yizhi >> >> 2018-03-07 11:11 GMT-08:00 Chris Olivier : >> > yeah I fixed that >> > >> > On Wed, Mar 7, 2018 at 11:09 AM, Marco de Abreu < >> > marco.g.ab...@googlemail.com> wrote: >> > >> >> Would it be possible to make tickets assigned to Nobody by default? >> >> >> >> On Wed, Mar 7, 2018 at 6:15 PM, Chris Olivier >> >> wrote: >> >> >> >> > Please assign JIRA issue to yourself after creating. I think it was >> >> > assigning to me for some reason. Just click "assign to me" link. >> >> > >> >> > If you don't have permissions, please email me directly (or even >> better, >> >> > ping on slack) and I will add you to a permission group (let me know >> if >> >> > you're a committer or contributor). >> >> > >> >> >> >> >> >> -- >> Yizhi Liu >> DMLC member >> Amazon Web Services >> Vancouver, Canada >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Assign JIRA issue
Hi Chris, could you help to add me to the permission group (committer)? Thanks, Yizhi 2018-03-07 11:11 GMT-08:00 Chris Olivier : > yeah I fixed that > > On Wed, Mar 7, 2018 at 11:09 AM, Marco de Abreu < > marco.g.ab...@googlemail.com> wrote: > >> Would it be possible to make tickets assigned to Nobody by default? >> >> On Wed, Mar 7, 2018 at 6:15 PM, Chris Olivier >> wrote: >> >> > Please assign JIRA issue to yourself after creating. I think it was >> > assigning to me for some reason. Just click "assign to me" link. >> > >> > If you don't have permissions, please email me directly (or even better, >> > ping on slack) and I will add you to a permission group (let me know if >> > you're a committer or contributor). >> > >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
[ANNOUNCE] Apache MXNet (incubating) 1.1.0 Release
Hello all, The Apache MXNet (incubating) Community announces the availability of Apache MXNet (incubating) 1.1.0! Apache MXNet (incubating) is a deep learning framework designed for both efficiency and flexibility. It allows you to mix symbolic and imperative programming to maximize efficiency and productivity. This release improves performance on GPU and CPU. With more operators supported, we are continuing to improve examples and tutorials. A full list of the changes in this release can be found in the release notes: https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.1.0+Release+Notes A Link to the Download is here: https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.1.0 If you prefer to build from source and experiment with various compile-time configuration options, use this link to get the instructions: http://mxnet.incubator.apache.org/install/index.html Or You can download and play with MXNet easily using one of the options below: 1. The Pip package can be found here: https://pypi.python.org/pypi/mxnet 2. The Docker Images can be found here: https://hub.docker.com/r/mxnet/python/ The release tag used for the 1.1.0 release is: https://github.com/apache/incubator-mxnet/tree/1.1.0 Some more MXNet Resources: 1. Issues: https://github.com/apache/incubator-mxnet/issues 2. Wiki: https://cwiki.apache.org/confluence/display/MXNET If you want to learn more about MXNet visit http://mxnet.incubator.apache.org/ Finally, you are welcome to join and also invite your friends to the dynamic and growing MXNet community by subscribing to dev@mxnet.incubator.apache.org Thanks! Apache MXNet (incubating) Team ___ DISCLAIMER: Apache MXNet (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
[RESULTS] [VOTE] Release MXNet version 1.1.0.RC1
This vote passes with 4 +1 votes (4 bindings) and no 0 or -1 votes. Binding +1: Tianqi Chen Nan Zhu Chris Olivier Xingjian Shi Vote thread: https://lists.apache.org/thread.html/500fb648e8629249b904d9fc56e7737c5312dfe54c17d4d5d06468a8@%3Cdev.mxnet.apache.org%3E I'll continue with the release process on general@ and the release announcement will follow in the next few days. -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: [VOTE] Release MXNet version 1.1.0.RC1
Remind that the voting will be closed tonight. Please help to verify whether the release candidate meets the standard and vote accordingly. Thanks, Yizhi 2018-02-10 19:30 GMT-08:00 Nan Zhu : > tested with scala-package's building and unit test > > +1 > > > > On Sat, Feb 10, 2018 at 10:49 AM, YiZhi Liu wrote: > >> Let's make the voting end at 11:10 p.m., Tuesday, February. 13th. >> >> Thanks for reminding. >> >> Best, >> Yizhi >> >> 2018-02-10 9:54 GMT-08:00 Marco de Abreu : >> > Please extend the vote until at least Tuesday, 13th of February due to >> the >> > weekend, thus counting Saturday and Sunday as half days. >> > >> > -Marco >> > >> > On Sat, Feb 10, 2018 at 6:09 PM, Tianqi Chen >> > wrote: >> > >> >> +1 >> >> >> >> tianqi >> >> On Fri, Feb 9, 2018 at 11:11 PM YiZhi Liu wrote: >> >> >> >> > Hi everyone, >> >> > >> >> > As we have updated the LICENSE file which caused the failure vote for >> >> > 1.1.0.RC0, I'd like to propose a vote to release Apache MXNet >> >> > (incubating) version 1.1.0.RC1. Voting will start now (Friday, Feb. 9) >> >> > and end at 11:10 p.m., Monday, Feb. 12. >> >> > >> >> > Link to release notes: >> >> > >> >> > https://cwiki.apache.org/confluence/display/MXNET/ >> >> Apache+MXNet+%28incubating%29+1.1.0+Release+Notes >> >> > >> >> > Link to release candidate 1.1.0.rc1: >> >> > https://github.com/apache/incubator-mxnet/releases/tag/1.1.0.rc1 >> >> > >> >> > PR which fixes the license: #9701 >> >> > >> >> > Voting results for 1.1.0.rc0: >> >> > >> >> > https://lists.apache.org/thread.html/a059cd379288b5781fc7251ed543f6 >> >> 4499248757ff2cb264be1bd6d7@%3Cdev.mxnet.apache.org%3E >> >> > >> >> > View this page and scroll down to "Build from Source" with source code >> >> > obtained from the 1.1.0.rc1 tag: >> >> > https://mxnet.incubator.apache.org/install/index.html >> >> > >> >> > Please remember to TEST first before voting accordingly: >> >> > +1 = approve >> >> > +0 = no opinion >> >> > -1 = disapprove (provide reason) >> >> > >> >> > Best, >> >> > Yizhi >> >> > >> >> > -- >> >> > Yizhi Liu >> >> > DMLC member >> >> > Amazon Web Services >> >> > Vancouver, Canada >> >> > >> >> >> >> >> >> -- >> Yizhi Liu >> DMLC member >> Amazon Web Services >> Vancouver, Canada >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: [VOTE] Release MXNet version 1.1.0.RC1
Let's make the voting end at 11:10 p.m., Tuesday, February. 13th. Thanks for reminding. Best, Yizhi 2018-02-10 9:54 GMT-08:00 Marco de Abreu : > Please extend the vote until at least Tuesday, 13th of February due to the > weekend, thus counting Saturday and Sunday as half days. > > -Marco > > On Sat, Feb 10, 2018 at 6:09 PM, Tianqi Chen > wrote: > >> +1 >> >> tianqi >> On Fri, Feb 9, 2018 at 11:11 PM YiZhi Liu wrote: >> >> > Hi everyone, >> > >> > As we have updated the LICENSE file which caused the failure vote for >> > 1.1.0.RC0, I'd like to propose a vote to release Apache MXNet >> > (incubating) version 1.1.0.RC1. Voting will start now (Friday, Feb. 9) >> > and end at 11:10 p.m., Monday, Feb. 12. >> > >> > Link to release notes: >> > >> > https://cwiki.apache.org/confluence/display/MXNET/ >> Apache+MXNet+%28incubating%29+1.1.0+Release+Notes >> > >> > Link to release candidate 1.1.0.rc1: >> > https://github.com/apache/incubator-mxnet/releases/tag/1.1.0.rc1 >> > >> > PR which fixes the license: #9701 >> > >> > Voting results for 1.1.0.rc0: >> > >> > https://lists.apache.org/thread.html/a059cd379288b5781fc7251ed543f6 >> 4499248757ff2cb264be1bd6d7@%3Cdev.mxnet.apache.org%3E >> > >> > View this page and scroll down to "Build from Source" with source code >> > obtained from the 1.1.0.rc1 tag: >> > https://mxnet.incubator.apache.org/install/index.html >> > >> > Please remember to TEST first before voting accordingly: >> > +1 = approve >> > +0 = no opinion >> > -1 = disapprove (provide reason) >> > >> > Best, >> > Yizhi >> > >> > -- >> > Yizhi Liu >> > DMLC member >> > Amazon Web Services >> > Vancouver, Canada >> > >> -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
[VOTE] Release MXNet version 1.1.0.RC1
Hi everyone, As we have updated the LICENSE file which caused the failure vote for 1.1.0.RC0, I'd like to propose a vote to release Apache MXNet (incubating) version 1.1.0.RC1. Voting will start now (Friday, Feb. 9) and end at 11:10 p.m., Monday, Feb. 12. Link to release notes: https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.1.0+Release+Notes Link to release candidate 1.1.0.rc1: https://github.com/apache/incubator-mxnet/releases/tag/1.1.0.rc1 PR which fixes the license: #9701 Voting results for 1.1.0.rc0: https://lists.apache.org/thread.html/a059cd379288b5781fc7251ed543f64499248757ff2cb264be1bd6d7@%3Cdev.mxnet.apache.org%3E View this page and scroll down to "Build from Source" with source code obtained from the 1.1.0.rc1 tag: https://mxnet.incubator.apache.org/install/index.html Please remember to TEST first before voting accordingly: +1 = approve +0 = no opinion -1 = disapprove (provide reason) Best, Yizhi -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Release Status - MXNet 1.1.0.RC0
Thanks Haibin. I am working on the RC1 release. Best, Yizhi 2018-02-08 11:08 GMT-08:00 Haibin Lin : > Hi all, > > The vote on general@ for 1.1.0.RC0 failed with 2 -1 votes (2 bindings) and > no 0 or +1 votes. > -1 votes (binding) and reasons: > Justin and Henri - LICENSE has issues > > Vote thread on general@: > https://lists.apache.org/thread.html/453663efa82e829e746ace79ece7b87209a856f7a7a6e3b2bfaf0d8f@%3Cgeneral.incubator.apache.org%3E > > PR #9701 (from @mbaijal) which updated the LICENSE file was approved by > mentors and merged yesterday. Another release candidate with a updated > LICENSE file will be proposed soon. > > Unfortunately, I will be traveling in China in the next three weeks. I'll > let Yizhi lead the remaining efforts for 1.1.0 release. > > Thanks, > Haibin -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: [VOTE] Release Apache MXNet (incubating) version 1.1.0.RC0
n < > > > > > > > haibin.lin@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > Given that most people on dev@ are in favor of a minor > > > > release > > > > > > > > (1.1.0) > > > > > > > > > > > instead of a patch release due to API changes, I'd like > > to > > > > > > propose > > > > > > > a > > > > > > > > > vote > > > > > > > > > > > to release Apache MXNet (incubating) 1.1.0. Voting will > > > start > > > > > now > > > > > > > > > > (Sunday, > > > > > > > > > > > January 28th) and end at 1pm Wednesday, January 31th > PST. > > > > > > > > > > > > > > > > > > > > > > Link to release notes: > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/ > > > > > > > > > > > Apache+MXNet+%28incubating%29+1.1.0+Release+Notes > > > > > > > > > > > > > > > > > > > > > > Link to release candidate 1.1.0.rc0: > > > > > > > > > > > https://github.com/apache/ > incubator-mxnet/releases/tag/ > > > > > 1.1.0.rc0 > > > > > > > > > > > > > > > > > > > > > > View this page and scroll down to “Build from Source” > > with > > > > > source > > > > > > > > code > > > > > > > > > > > obtained from the 1.1.0.rc0 tag: > > > > > > > > > > > https://mxnet.incubator.apache.org/install/index.html > > > > > > > > > > > > > > > > > > > > > > (Note: The README.md points to the 1.1.0 tag and does > not > > > > work > > > > > at > > > > > > > the > > > > > > > > > > > moment.) > > > > > > > > > > > > > > > > > > > > > > Please remember to TEST first before voting > accordingly: > > > > > > > > > > > +1 = approve > > > > > > > > > > > +0 = no opinion > > > > > > > > > > > -1 = disapprove (provide reason) > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Haibin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Sandeep Krishnamurthy > > > > > > > > > > > > > > > > > > > > > > > > -- > Sandeep Krishnamurthy > > -- > Yizhi Liu > DMLC member > Amazon Web Services > Vancouver, Canada > >
Re: Guidance For Posting Jar to Apache Maven Repo
Thank you guys, looks like we can post using Apache ID. No additional registration is required. 2018-01-18 14:23 GMT-08:00 Nan Zhu : > well, something can be borrowed from Spark again: > https://github.com/apache/spark/blob/master/dev/create- > release/release-build.sh > > Best, > > Nan > > On Wed, Jan 17, 2018 at 6:01 PM, Naveen Swamy wrote: > > > Hi Yizhi, > > Here is the Apache guideline > > http://www.apache.org/dev/publishing-maven-artifacts.html. > > > > -Naveen > > > > > > > > On Wed, Jan 17, 2018 at 5:59 PM, YiZhi Liu wrote: > > > > > Since we now have changed Scala package prefix to org.apache, does > > someone > > > have guidance for posting it to Apache's Maven Repository? > > > > > > FYI, ml.dmlc.mxnet was posted to oss.sonatype.org. > > > > > > -- > > > Yizhi Liu > > > DMLC member > > > Amazon Web Services > > > Vancouver, Canada > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Guidance For Posting Jar to Apache Maven Repo
Since we now have changed Scala package prefix to org.apache, does someone have guidance for posting it to Apache's Maven Repository? FYI, ml.dmlc.mxnet was posted to oss.sonatype.org. -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
[jira] [Commented] (MXNET-4) Performance: Refactor Random and ParallelRandom resources to use MKL for MKL builds
[ https://issues.apache.org/jira/browse/MXNET-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327556#comment-16327556 ] Yizhi Liu commented on MXNET-4: --- I think it's easier to have Random support MKL since they both design to fulfill an array. > Performance: Refactor Random and ParallelRandom resources to use MKL for MKL > builds > --- > > Key: MXNET-4 > URL: https://issues.apache.org/jira/browse/MXNET-4 > Project: Apache MXNet > Issue Type: Improvement >Reporter: Chris Olivier >Priority: Major > > Refactor Random and ParallelRandom resources to use MKL for MKL builds > Things such as RngUniform, etc. Similarly to what is done for dropout > operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Refactoring MXNet scala code to use "org.apache.mxnet"
Since we now have changed package prefix to org.apache, does someone have guidance for posting it to Apache's Maven Repository? ml.dmlc.mxnet was posted to oss.sonatype.org. 2018-01-04 17:14 GMT-08:00 Lupesko, Hagay : > Suneel, > > I tend to think for this issue, GitHub issue is good enough and we do not > need JIRA. > Can you clarify what is the advantage you see in using JIRA over GitHub > issue for this specific case? > > Thanks! > Hagay > > On 1/4/18, 16:34, "Suneel Marthi" wrote: > > Jira has been around for a while - > https://issues.apache.org/jira/projects/MXNET/ > > switch to using jira > > On Thu, Jan 4, 2018 at 7:31 PM, Roshani Nagmote < > roshaninagmo...@gmail.com> > wrote: > > > Hi, > > > > As currently, MXNet does not have Jira project, I have created > github issue > > for now. > > https://github.com/apache/incubator-mxnet/issues/9315 > > > > Will create the PR and link the issue there. > > > > Thanks, > > Roshani > > > > On Thu, Jan 4, 2018 at 3:08 PM, Naveen Swamy > wrote: > > > > > Hi Suneel, > > > > > > Did we decide that we will using Jira going forward? If not, can > someone > > > summarize on the improvement email on the consensus and lets make > it > > > universal and how to use it, what is expected, etc., > > > > > > For the record, I like the idea of using Jira for more openness. > > > > > > Also, MXNet does not have Jira project, can you help creating one? > > > > > > Thanks, Naveen > > > > > > > > > On Thu, Jan 4, 2018 at 2:35 PM, Suneel Marthi > > wrote: > > > > > > > Is there a Jira for this? Please create a Jira and reference > that in > > the > > > PR > > > > for this. > > > > > > > > On Thu, Jan 4, 2018 at 5:16 PM, Roshani Nagmote < > > > roshaninagmo...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hello all, > > > > > > > > > > I am working on publishing mxnet-scala release to maven > repository > > and > > > > as a > > > > > part of that, I will also be refactoring mxnet-scala > > code/tests/example > > > > and > > > > > docs to use "org.apache.mxnet" instead of "ml.dmlc.mxnet". > > > > > > > > > > Currently, MXNet-Scala > > > > > <http://mxnet.incubator.apache.org/api/scala/index.html> > library > > uses > > > > > "ml.dmlc.mxnet" packages. This work will change the way to > import > > > modules > > > > > when using mxnet-scala package. > > > > > > > > > > *Old way:* > > > > > > > > > > scala> import ml.dmlc.mxnet._ > > > > >import ml.dmlc.mxnet._scala> val arr = NDArray.ones(2, 3) > > > > >arr: ml.dmlc.mxnet.NDArray = ml.dmlc.mxnet.NDArray@f5e74790 > > > > > > > > > > *New way:* > > > > > > > > > > scala> import org.apache.mxnet._ > > > > >import org.apache.mxnet._ > > > > > scala> val arr = NDArray.ones(2, 3) > > > > >arr: org.apache.mxnet.NDArray = org.apache.mxnet.NDArray@ > f5e74790 > > > > > > > > > > > > > > > Please let me know if anyone has any thoughts or issues with > this > > > change. > > > > > > > > > > Thanks, > > > > > Roshani > > > > > > > > > > > > > > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Module maintainers proposal
+1 for adding @CodingCat (Nan Zhu) as owner of Scala package. 2018-01-08 21:58 GMT-08:00 Mu Li : > It has been a while for discussing having maintainers for individual > modules. A maintainer for a module will be the main person who reviews and > approves PRs contributed to that module. > > Currently, some maintainers are listed in the file CODEOWNERS > <https://github.com/apache/incubator-mxnet/blob/master/CODEOWNERS>: > > # Owners of Apache MXNet > # Global owners > *@apache/mxnet-committers > # Owners of language bindings > R-package/*@thirdwing > scala-package/*@javelinjs > perl-package/*@sergeykolychev > # CMake owners > CMakeLists.txt@cjolivier01 > cmake/*@cjolivier01 > > However, this list is incomplete. This document proposes how to partition > mxnet codes into modules and put tentative maintainers for each module. > >- I tried to select the activate contributor who contributed most to the >module, not the new contributor who is going to work for it. But I may > be >outdated. >- Code review is not easy. So maintainers may need help at the >beginning. >- Eric (@piiswrong) did most PR reviews, so I didn't put his name on >every module. > > Any suggestion is welcome. > FrontendPYTHON: @szha > > python/ > > R: @thirdwing @hetong007 > > R-package/ > > SCALA: @CodingCat @javelinjs > > scala-package/ > > PERL@sergeykolychev > > perl-package/ > > C++? > > cpp-package/ > > MATLAB: DEPRECATE IT? > > matlab/ > > AMALGAMATION: DEPRECATE IT? > > amalgamation/ > > Backend@reminisce @eric-haibin-lin > > include/ > src/ > > Build@cjolivier01 > > docker/ > docker_multiarch/ > make/ > Makefile > cmake/ > CMakeLists.txt > setup-utils/ > prepare_mkl.sh > > Test@marcoabreu > > tests/ > Jenkinsfile > > Doc & Website@kevinthesun > > docs/ > > ExamplesNot sure, since we have a lot of contributors here. We probably > need to remove low quality examples and assign one maintainer for each of > the rest. > > example/ > > ToolsNot sure as well, since we have a bunch of things there. Probably need > to the same thing as examples > > tools/ > plugin/ > > AppendixLines of codes added into each folder on the last two months: > Lines of codes added into example/ > Lines of codes added into src/ > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: request for reviewing PR in ps-lite
to my understanding, this is not about fault-tolerance, i.e., restart when worker/server fail, right? I can help to review. ping @Mu for advice. 2017-12-05 13:13 GMT-08:00 CodingCat : > ping > > On Sat, Dec 2, 2017 at 10:04 AM, CodingCat wrote: > > > ping > > > > On Fri, Dec 1, 2017 at 12:18 AM, Nan Zhu wrote: > > > >> Hi, all > >> > >> I have been working on integrating MXNet with Spark in a more > >> full-fledged manner. > >> > >> One of the most critical pre-conditions is to make parameter server in > >> mxnet support multiple workers per process. I created the PR in > >> https://github.com/dmlc/ps-lite/pull/121 (OK, sorry for being lateI > >> should have finished it earlier) > >> > >> This PR includes some refactoring of those too long methods, to > highlight > >> the changes > >> > >> 1. https://github.com/dmlc/ps-lite/pull/112 includes the changes > related > >> to refactoring > >> > >> 2. https://github.com/CodingCat/ps-lite/pull/3/files includes the > >> changes related to the key functionality > >> > >> 3. https://github.com/dmlc/ps-lite/pull/121 contains everything (Please > >> review this one) > >> > >> > >> I am not sure who is the current owner of ps-lite, please help to share > >> your thoughts on the implementation. Only after this PR is merged and > >> ps-lite version is synced in mxnet repo, I can file the successive PRs > in > >> mxnet > >> > >> Thank you very much! > >> > >> Nan > >> > > > > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Protected master needs to be turned off
+1 2017-11-20 11:47 GMT-08:00 Tianqi Chen : > +1 until new CI is implemented. > > Tianqi > > On Mon, Nov 20, 2017 at 11:11 AM, Eric Xie wrote: > >> A lot of people seems to be confused, so let's clarify the separation of >> roles/responsibilities: >> >> 1. The committers that merge code are responsible for code quality and >> tests passing on master. >> >> 2. The CI / infra maintainers are responsible for keeping CI running >> properly and honestly reporting bugs. >> If test fails because Jenkins is faulty or slow, you need to fix Jenkins. >> If test fails because there is a bug in the code, then you did a good job >> catching bugs. It is not your responsibility to fix the bug. You merely >> should report it to committers. >> >> Thanks, >> Junyuan Xie >> >> >> On 2017-11-19 16:07, Gautam wrote: >> > -1 >> > >> > Please see inline. >> > >> > On Nov 19, 2017 12:51 PM, "Eric Xie" wrote: >> > >> > Hi all, >> > I'm starting this thread to vote on turning off protected master. The >> > reasons are: >> > >> > 1. Since we turned on protected master pending PRs has grown from 40 to >> 80. >> > It is severely slowing down development. >> > >> > >> > Turning protection off, will give you ability to merge the code >> which >> > has build failure. How and more importantly who will be best judge to >> > figure out what's is wrong ? If CI is a problem then we have people from >> > infra team, who are sort of 'on call" day and night trying to fix. >> > Including me trying to fix the slave at 2 am in night. I have seen >> > commiters sending PRs and if it fails without even looking at the >> > reason,which could be just a temporary error, they reach out to infra >> team >> > immediately. So I don't agree that we should turn off the protected >> branch >> > just for the sake of speed. We should care more on quality than speed. >> > >> > >> > 2. Committers, not CI, are ultimately responsible for the code they >> merge. >> > You should only override the CI when you are very confident that CI is >> the >> > problem, not your code. If it turns out you are wrong, you should fix it >> > ASAP. This is the bare minimum requirement for all committers: BE >> > RESPONSIBLE. >> > >> > How do we make sure that this in deed happens? >> > >> > I'm aware of the argument for using protected master: It make sure that >> >> >> >> > master is stable. >> > >> > Well, master will be most stable if we stop adding any commits to it. But >> > that's not what we want is it? >> > >> > No we are not saying don't add anything in master, we are just saying >> > please don't add bad code to the master. And yes I have seen bad code has >> > been merged to the master when protected branch was not enabled. >> > >> > Protected master hardly adds any stability. The faulty tests that breaks >> > master at random got merged into master because they happened to succeed >> > once. >> > >> > That's not true, it filter out one of the important aspect that at least >> > when code was merged it completed the whole cycle of build and test. Sure >> > flaky test we can track down. >> > >> > Thanks, >> > Junyuan Xie >> > >> -- Yizhi Liu DMLC member
MXNet Slack channel
Requesting an invitation to Slack. Thanks, Yizhi -- Yizhi Liu DMLC member
Re: what is NDArrayFuncReturn ?
This is a internal class for wrapping NDArray(s) returned by function calls. It handles functions that return more than one array, and tracks the array dependencies. You can call get() or apply() to get NDArray 2017-11-05 11:55 GMT+08:00 TongKe Xue : > Hi, > > 1. I'm running into issues with NDArrayFuncReturn vs NDArray. In particular: > > 1a. Transpose is defined on NDArray, but does not appear to be defined > on NDArrayFuncReturn: > https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/ml/dmlc/mxnet/NDArray.scala#L665 > > 2. After calling broadcast_minus , I get a NDArrayFuncReturn > > 3. There are functions 'head, get, apply' for doing NDArrayFuncReturn > -> NDArray: > https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/ml/dmlc/mxnet/NDArray.scala#L1099-L1104 > > So: > * What is NDArrayFuncReturn ? > * If I'm getting NDArrayFuncReturn (a private class) returned, am I > calling some internal API I should not be using ? > * What is correct way o do NDArrayFuncReturn -> NDArray ? > > Thanks, > --TongKe -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: mxnet java prebuilt
Hi TongKe, I've setup a travis build for scala-package, here (https://oss.sonatype.org/#nexus-search;quick~mxnet-full_2.11) you can find mxnet linux cu70, cu80, cu90, etc. We'll make full release (instead of SNAPSHOT) when the build system is stable. Best, Yizhi 2017-10-30 4:15 GMT+08:00 TongKe Xue : > Hi, > > Looking at https://mvnrepository.com/artifact/ml.dmlc.mxnet?sort=newest > it appears there are no new maven packages since Jun 3 2017. > > For using the mxnet scala api, is the standard solution to: > > 1. always build from github / source via > https://mxnet.incubator.apache.org/get_started/build_from_source.html > > or > > 2. there's some other repo to use? > > > Thanks, > --TongKe -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: Does mxnet NDArray have autodiff ?
Currently, it hasn't. We'll add soon. 2017-10-20 17:43 GMT+08:00 TongKe Xue : > Hi! > > [I'm referring to the Scala API] > > 1. mxnet symbol has auto diff: > https://mxnet.incubator.apache.org/tutorials/r/symbol.html > > 2. from Googling around, I can't find NDArray auto diff anywhere > > 3. however, NDArray tracks dependencies -- which, in theory, should > be enough for doing autodiff > > Does NDArray have autodiff somewhere? > > Thanks, > --TongKe -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: mxnet Scala Convolution
You can retrieve the functions/operators by reflection. Or, when compiling the scala package, it prints all operators out. 2017-10-19 8:15 GMT+00:00 YiZhi Liu : > The javadoc cannot be added in this way. I'm afraid we have to write > another 'javadoc generator' for these operators. > > 2017-10-19 7:53 GMT+00:00 TongKe Xue : >> HI YiZhi, >> >> >> 1. I looked at: >> scala-package/macros/src/main/scala/ml/dmlc/mxnet/SymbolMacro.scala >> scala-package/macros/src/main/scala/ml/dmlc/mxnet/NDArrayMacro.scala >> >> These files appear to dynamically register nnvm functions into >> NDArray/Symbol right? >> >> I would like the "javadocs" of the dynamically added functions. >> >> >> 2. I looked at mxnet-macros_2.11-0.11.0-SNAPSHOT-javadoc.jar >> >> That jar looks mostly empty. Is that expected, or am I doing something wrong? >> >> >> 3. Currently, the best way I know for retrieving all the functions is >> to fire up a repl and run reflection. Is there a better method? >> >> >> Thanks, >> --TongKe >> >> >> >> On Wed, Oct 18, 2017 at 8:59 PM, YiZhi Liu wrote: >>> Hi TongKe, >>> >>> The symbols you are looking for are auto-generated by scala macros. >>> Pls refer to scala-package/macros >>> >>> 2017-10-19 0:40 GMT+00:00 TongKe Xue : >>>> Hi Rahul, >>>> >>>> Thanks for explaining the high level design + pointing to the >>>> implementation details. >>>> >>>> Besides reading the C++ code and mentally translating the Scala >>>> calls, is there a way to get a list of all generated Scala functions? >>>> >>>> I have looked at: >>>> >>>> 1. https://mxnet.incubator.apache.org/api/scala/symbol.html >>>> shows a few examples, but is not exhaustive >>>> >>>> 2. >>>> https://mxnet.incubator.apache.org/api/scala/docs/index.html#ml.dmlc.mxnet.Symbol >>>> appears more comprehensive, but I find neither Convolution nor Softmax >>>> there. >>>> >>>> >>>> More specifically, my question is: nnvm adds a bunch of Scala bindings >>>> to C++ code. How do I get a list of all these bindings (name, type of >>>> inputs, type of output). >>>> >>>> >>>> Thanks! >>>> --TongKe >>>> >>>> >>>> On Wed, Oct 18, 2017 at 5:28 PM, Rahul Huilgol >>>> wrote: >>>>> Hi TongKe, >>>>> >>>>> These are operators defined in the c++ backend under src/operator. For >>>>> example convolution is here >>>>> https://github.com/apache/incubator-mxnet/blob/master/src/operator/convolution.cc >>>>> . The operators are registered using nnvm, which helps automatically >>>>> generate the frontend functions. >>>>> >>>>> This tutorial on how to add a backend operator >>>>> <https://github.com/apache/incubator-mxnet/blob/master/docs/how_to/add_op_in_backend.md> >>>>> contains information on how to register such operators, which would help >>>>> you understand the above file. >>>>> An excerpt from there (for quadratic operator) : "If you use python, when >>>>> you type import mxnet as mx, two python functions for invoking your >>>>> backend >>>>> implementation are generated on the fly: one is for imperative programming >>>>> registered as mxnet.ndarray.quadratic or mx.nd.quadratic for short; the >>>>> other one is for symbolic programming registered under module >>>>> mxnet.symbol.quadratic or mx.sym.quadratic for short." >>>>> >>>>> I'd think the Scala package works similarly. >>>>> >>>>> Regards, >>>>> Rahul >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Oct 18, 2017 at 5:06 PM, TongKe Xue wrote: >>>>> >>>>>> My earlier question was a bit messy. >>>>>> >>>>>> To rephrase my question: >>>>>> >>>>>> 1. Scala AlexNet sample code calls Symbol.Convolution: >>>>>> >>>>>> https://github.com/apache/incubator-mxnet/blob/master/ >>>>>> scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/visualization/ >>>>>> AlexNet.scala#L30 >>>>>
Re: mxnet Scala Convolution
You can retrieve the functions/operators by reflection. Or, when compiling the scala package, it prints all operators out. 2017-10-19 8:15 GMT+00:00 YiZhi Liu : > The javadoc cannot be added in this way. I'm afraid we have to write > another 'javadoc generator' for these operators. > > 2017-10-19 7:53 GMT+00:00 TongKe Xue : >> HI YiZhi, >> >> >> 1. I looked at: >> scala-package/macros/src/main/scala/ml/dmlc/mxnet/SymbolMacro.scala >> scala-package/macros/src/main/scala/ml/dmlc/mxnet/NDArrayMacro.scala >> >> These files appear to dynamically register nnvm functions into >> NDArray/Symbol right? >> >> I would like the "javadocs" of the dynamically added functions. >> >> >> 2. I looked at mxnet-macros_2.11-0.11.0-SNAPSHOT-javadoc.jar >> >> That jar looks mostly empty. Is that expected, or am I doing something wrong? >> >> >> 3. Currently, the best way I know for retrieving all the functions is >> to fire up a repl and run reflection. Is there a better method? >> >> >> Thanks, >> --TongKe >> >> >> >> On Wed, Oct 18, 2017 at 8:59 PM, YiZhi Liu wrote: >>> Hi TongKe, >>> >>> The symbols you are looking for are auto-generated by scala macros. >>> Pls refer to scala-package/macros >>> >>> 2017-10-19 0:40 GMT+00:00 TongKe Xue : >>>> Hi Rahul, >>>> >>>> Thanks for explaining the high level design + pointing to the >>>> implementation details. >>>> >>>> Besides reading the C++ code and mentally translating the Scala >>>> calls, is there a way to get a list of all generated Scala functions? >>>> >>>> I have looked at: >>>> >>>> 1. https://mxnet.incubator.apache.org/api/scala/symbol.html >>>> shows a few examples, but is not exhaustive >>>> >>>> 2. >>>> https://mxnet.incubator.apache.org/api/scala/docs/index.html#ml.dmlc.mxnet.Symbol >>>> appears more comprehensive, but I find neither Convolution nor Softmax >>>> there. >>>> >>>> >>>> More specifically, my question is: nnvm adds a bunch of Scala bindings >>>> to C++ code. How do I get a list of all these bindings (name, type of >>>> inputs, type of output). >>>> >>>> >>>> Thanks! >>>> --TongKe >>>> >>>> >>>> On Wed, Oct 18, 2017 at 5:28 PM, Rahul Huilgol >>>> wrote: >>>>> Hi TongKe, >>>>> >>>>> These are operators defined in the c++ backend under src/operator. For >>>>> example convolution is here >>>>> https://github.com/apache/incubator-mxnet/blob/master/src/operator/convolution.cc >>>>> . The operators are registered using nnvm, which helps automatically >>>>> generate the frontend functions. >>>>> >>>>> This tutorial on how to add a backend operator >>>>> <https://github.com/apache/incubator-mxnet/blob/master/docs/how_to/add_op_in_backend.md> >>>>> contains information on how to register such operators, which would help >>>>> you understand the above file. >>>>> An excerpt from there (for quadratic operator) : "If you use python, when >>>>> you type import mxnet as mx, two python functions for invoking your >>>>> backend >>>>> implementation are generated on the fly: one is for imperative programming >>>>> registered as mxnet.ndarray.quadratic or mx.nd.quadratic for short; the >>>>> other one is for symbolic programming registered under module >>>>> mxnet.symbol.quadratic or mx.sym.quadratic for short." >>>>> >>>>> I'd think the Scala package works similarly. >>>>> >>>>> Regards, >>>>> Rahul >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Oct 18, 2017 at 5:06 PM, TongKe Xue wrote: >>>>> >>>>>> My earlier question was a bit messy. >>>>>> >>>>>> To rephrase my question: >>>>>> >>>>>> 1. Scala AlexNet sample code calls Symbol.Convolution: >>>>>> >>>>>> https://github.com/apache/incubator-mxnet/blob/master/ >>>>>> scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/visualization/ >>>>>> AlexNet.scala#L30 >>>>>
Re: mxnet Scala Convolution
The javadoc cannot be added in this way. I'm afraid we have to write another 'javadoc generator' for these operators. 2017-10-19 7:53 GMT+00:00 TongKe Xue : > HI YiZhi, > > > 1. I looked at: > scala-package/macros/src/main/scala/ml/dmlc/mxnet/SymbolMacro.scala > scala-package/macros/src/main/scala/ml/dmlc/mxnet/NDArrayMacro.scala > > These files appear to dynamically register nnvm functions into > NDArray/Symbol right? > > I would like the "javadocs" of the dynamically added functions. > > > 2. I looked at mxnet-macros_2.11-0.11.0-SNAPSHOT-javadoc.jar > > That jar looks mostly empty. Is that expected, or am I doing something wrong? > > > 3. Currently, the best way I know for retrieving all the functions is > to fire up a repl and run reflection. Is there a better method? > > > Thanks, > --TongKe > > > > On Wed, Oct 18, 2017 at 8:59 PM, YiZhi Liu wrote: >> Hi TongKe, >> >> The symbols you are looking for are auto-generated by scala macros. >> Pls refer to scala-package/macros >> >> 2017-10-19 0:40 GMT+00:00 TongKe Xue : >>> Hi Rahul, >>> >>> Thanks for explaining the high level design + pointing to the >>> implementation details. >>> >>> Besides reading the C++ code and mentally translating the Scala >>> calls, is there a way to get a list of all generated Scala functions? >>> >>> I have looked at: >>> >>> 1. https://mxnet.incubator.apache.org/api/scala/symbol.html >>> shows a few examples, but is not exhaustive >>> >>> 2. >>> https://mxnet.incubator.apache.org/api/scala/docs/index.html#ml.dmlc.mxnet.Symbol >>> appears more comprehensive, but I find neither Convolution nor Softmax >>> there. >>> >>> >>> More specifically, my question is: nnvm adds a bunch of Scala bindings >>> to C++ code. How do I get a list of all these bindings (name, type of >>> inputs, type of output). >>> >>> >>> Thanks! >>> --TongKe >>> >>> >>> On Wed, Oct 18, 2017 at 5:28 PM, Rahul Huilgol >>> wrote: >>>> Hi TongKe, >>>> >>>> These are operators defined in the c++ backend under src/operator. For >>>> example convolution is here >>>> https://github.com/apache/incubator-mxnet/blob/master/src/operator/convolution.cc >>>> . The operators are registered using nnvm, which helps automatically >>>> generate the frontend functions. >>>> >>>> This tutorial on how to add a backend operator >>>> <https://github.com/apache/incubator-mxnet/blob/master/docs/how_to/add_op_in_backend.md> >>>> contains information on how to register such operators, which would help >>>> you understand the above file. >>>> An excerpt from there (for quadratic operator) : "If you use python, when >>>> you type import mxnet as mx, two python functions for invoking your backend >>>> implementation are generated on the fly: one is for imperative programming >>>> registered as mxnet.ndarray.quadratic or mx.nd.quadratic for short; the >>>> other one is for symbolic programming registered under module >>>> mxnet.symbol.quadratic or mx.sym.quadratic for short." >>>> >>>> I'd think the Scala package works similarly. >>>> >>>> Regards, >>>> Rahul >>>> >>>> >>>> >>>> >>>> On Wed, Oct 18, 2017 at 5:06 PM, TongKe Xue wrote: >>>> >>>>> My earlier question was a bit messy. >>>>> >>>>> To rephrase my question: >>>>> >>>>> 1. Scala AlexNet sample code calls Symbol.Convolution: >>>>> >>>>> https://github.com/apache/incubator-mxnet/blob/master/ >>>>> scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/visualization/ >>>>> AlexNet.scala#L30 >>>>> >>>>> 2. Symbol.scala does not contain the string "Convolution" >>>>> >>>>> https://github.com/apache/incubator-mxnet/blob/master/ >>>>> scala-package/core/src/main/scala/ml/dmlc/mxnet/Symbol.scala#L982 >>>>> >>>>> Question: where/how is Symbol.Convolution defined? >>>>> >>>>> On Wed, Oct 18, 2017 at 4:10 PM, TongKe Xue wrote: >>>>> > Hi, >>>>> > >>>>> > I am reading: https://mxnet.incubator.apache.org/api/scala/symbol.html >>>>> > >>>>> > I see Symbol.Variable, Symbol.Convolution >>>>> > >>>>> > When I look at Symbol.scala, I see Symbol.Variable at: >>>>> > https://github.com/apache/incubator-mxnet/blob/master/ >>>>> scala-package/core/src/main/scala/ml/dmlc/mxnet/Symbol.scala#L982 >>>>> > >>>>> > However, I can't find where Convolution, SoftMax, FullyConnected, ... >>>>> > are defined. >>>>> > >>>>> > Where are these Symbols defined? >>>>> > >>>>> > (I have also tried: grep "Convolution" . -R | grep scala | grep def -- >>>>> > but found nothing). >>>>> > >>>>> > Thanks, >>>>> > --TongKe >>>>> >>>> >>>> >>>> >>>> -- >>>> Rahul Huilgol >> >> >> >> -- >> Yizhi Liu >> DMLC member >> Technical Manager >> Qihoo 360 Inc, Shanghai, China -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: mxnet Scala Convolution
Hi TongKe, The symbols you are looking for are auto-generated by scala macros. Pls refer to scala-package/macros 2017-10-19 0:40 GMT+00:00 TongKe Xue : > Hi Rahul, > > Thanks for explaining the high level design + pointing to the > implementation details. > > Besides reading the C++ code and mentally translating the Scala > calls, is there a way to get a list of all generated Scala functions? > > I have looked at: > > 1. https://mxnet.incubator.apache.org/api/scala/symbol.html > shows a few examples, but is not exhaustive > > 2. > https://mxnet.incubator.apache.org/api/scala/docs/index.html#ml.dmlc.mxnet.Symbol > appears more comprehensive, but I find neither Convolution nor Softmax there. > > > More specifically, my question is: nnvm adds a bunch of Scala bindings > to C++ code. How do I get a list of all these bindings (name, type of > inputs, type of output). > > > Thanks! > --TongKe > > > On Wed, Oct 18, 2017 at 5:28 PM, Rahul Huilgol wrote: >> Hi TongKe, >> >> These are operators defined in the c++ backend under src/operator. For >> example convolution is here >> https://github.com/apache/incubator-mxnet/blob/master/src/operator/convolution.cc >> . The operators are registered using nnvm, which helps automatically >> generate the frontend functions. >> >> This tutorial on how to add a backend operator >> <https://github.com/apache/incubator-mxnet/blob/master/docs/how_to/add_op_in_backend.md> >> contains information on how to register such operators, which would help >> you understand the above file. >> An excerpt from there (for quadratic operator) : "If you use python, when >> you type import mxnet as mx, two python functions for invoking your backend >> implementation are generated on the fly: one is for imperative programming >> registered as mxnet.ndarray.quadratic or mx.nd.quadratic for short; the >> other one is for symbolic programming registered under module >> mxnet.symbol.quadratic or mx.sym.quadratic for short." >> >> I'd think the Scala package works similarly. >> >> Regards, >> Rahul >> >> >> >> >> On Wed, Oct 18, 2017 at 5:06 PM, TongKe Xue wrote: >> >>> My earlier question was a bit messy. >>> >>> To rephrase my question: >>> >>> 1. Scala AlexNet sample code calls Symbol.Convolution: >>> >>> https://github.com/apache/incubator-mxnet/blob/master/ >>> scala-package/examples/src/main/scala/ml/dmlc/mxnetexamples/visualization/ >>> AlexNet.scala#L30 >>> >>> 2. Symbol.scala does not contain the string "Convolution" >>> >>> https://github.com/apache/incubator-mxnet/blob/master/ >>> scala-package/core/src/main/scala/ml/dmlc/mxnet/Symbol.scala#L982 >>> >>> Question: where/how is Symbol.Convolution defined? >>> >>> On Wed, Oct 18, 2017 at 4:10 PM, TongKe Xue wrote: >>> > Hi, >>> > >>> > I am reading: https://mxnet.incubator.apache.org/api/scala/symbol.html >>> > >>> > I see Symbol.Variable, Symbol.Convolution >>> > >>> > When I look at Symbol.scala, I see Symbol.Variable at: >>> > https://github.com/apache/incubator-mxnet/blob/master/ >>> scala-package/core/src/main/scala/ml/dmlc/mxnet/Symbol.scala#L982 >>> > >>> > However, I can't find where Convolution, SoftMax, FullyConnected, ... >>> > are defined. >>> > >>> > Where are these Symbols defined? >>> > >>> > (I have also tried: grep "Convolution" . -R | grep scala | grep def -- >>> > but found nothing). >>> > >>> > Thanks, >>> > --TongKe >>> >> >> >> >> -- >> Rahul Huilgol -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: What's everyone working on?
Hi Dominic, I'm working on 0.11-snapshot and we will soon have one. While the stable release will be after that we change package name from 'ml.dmlc' to 'org.apache'. 2017-09-27 0:04 GMT+08:00 Dominic Divakaruni : > That's great, YiZhi. Workday uses the Scala package and was looking for a > maven distro for v0.11. When do you think you'll have one up? > > On Tue, Sep 26, 2017 at 8:58 AM, YiZhi Liu wrote: > >> I'm currently working on maven deploy for scala package. >> >> 2017-09-26 16:00 GMT+08:00 Zihao Zheng : >> > I’m working on standalone TensorBoard, https://github.com/dmlc/ >> tensorboard <https://github.com/dmlc/tensorboard>, currently we’ve >> support several features in original TensorBoard from TensorFlow in pure >> Python without any DL framework dependency. >> > >> > Recently I’m trying to bring more features to this standalone version, >> but seems not very trivial as it depends on TensorFlow. Any advice are >> welcomed and looking for help. >> > >> > Thanks, >> > Zihao >> > >> >> 在 2017年9月26日，下午1:58，sandeep krishnamurthy >> 写道： >> >> >> >> I am currently working with Jiajie Chen (https://github.com/jiajiechen/) >> on >> >> building an automated periodic benchmarking framework to run various >> >> standard MXNet training jobs with both Symbolic and Gluon interface. >> This >> >> framework will run following standard training jobs on a nightly and >> weekly >> >> basis helping us to track performance improvements or regression early >> in >> >> the development cycle of MXNet. Both CPU and GPU instances are used >> >> capturing various metrics like training accuracy, validation accuracy, >> >> convergence, memory consumption, speed. >> >> >> >> To start with, we will be running Resnet50, Resnet152 on CIFAR and >> >> Synthetic Dataset. And, few more RNN and Bidirectional LSTM training >> jobs. >> >> >> >> Thanks, >> >> Sandeep >> >> >> >> >> >> On Mon, Sep 25, 2017 at 8:00 PM, Henri Yandell >> wrote: >> >> >> >>> Getting an instance of github.com/amzn/oss-dashboard setup for mxnet. >> >>> >> >>> Hopefully useful to write custom metric analysis; like: "most pull >> requests >> >>> from non-committer" and "PRs without committer comment". >> >>> >> >>> Hen >> >>> >> >>> On Mon, Sep 25, 2017 at 11:24 Seb Kiureghian >> wrote: >> >>> >> >>>> Hey dev@, >> >>>> >> >>>> In the spirit of bringing more activity to the mailing lists and >> growing >> >>>> the community, can everyone who is working on MXNet please share what >> >>>> you're working on? >> >>>> >> >>>> I'm working on >> >>>> -Redesigning the website >> >>>> <https://mxnet.incubator.apache.org/versions/master/index.html>. >> >>>> -Setting up a forum for user support. >> >>>> >> >>>> Seb Kiureghian >> >>>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Sandeep Krishnamurthy >> > >> >> >> >> -- >> Yizhi Liu >> DMLC member >> Technical Manager >> Qihoo 360 Inc, Shanghai, China >> > > > > -- > > > Dominic Divakaruni > 206.475.9200 Cell -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: What's everyone working on?
I'm currently working on maven deploy for scala package. 2017-09-26 16:00 GMT+08:00 Zihao Zheng : > I’m working on standalone TensorBoard, https://github.com/dmlc/tensorboard > <https://github.com/dmlc/tensorboard>, currently we’ve support several > features in original TensorBoard from TensorFlow in pure Python without any > DL framework dependency. > > Recently I’m trying to bring more features to this standalone version, but > seems not very trivial as it depends on TensorFlow. Any advice are welcomed > and looking for help. > > Thanks, > Zihao > >> 在 2017年9月26日，下午1:58，sandeep krishnamurthy 写道： >> >> I am currently working with Jiajie Chen (https://github.com/jiajiechen/) on >> building an automated periodic benchmarking framework to run various >> standard MXNet training jobs with both Symbolic and Gluon interface. This >> framework will run following standard training jobs on a nightly and weekly >> basis helping us to track performance improvements or regression early in >> the development cycle of MXNet. Both CPU and GPU instances are used >> capturing various metrics like training accuracy, validation accuracy, >> convergence, memory consumption, speed. >> >> To start with, we will be running Resnet50, Resnet152 on CIFAR and >> Synthetic Dataset. And, few more RNN and Bidirectional LSTM training jobs. >> >> Thanks, >> Sandeep >> >> >> On Mon, Sep 25, 2017 at 8:00 PM, Henri Yandell wrote: >> >>> Getting an instance of github.com/amzn/oss-dashboard setup for mxnet. >>> >>> Hopefully useful to write custom metric analysis; like: "most pull requests >>> from non-committer" and "PRs without committer comment". >>> >>> Hen >>> >>> On Mon, Sep 25, 2017 at 11:24 Seb Kiureghian wrote: >>> >>>> Hey dev@, >>>> >>>> In the spirit of bringing more activity to the mailing lists and growing >>>> the community, can everyone who is working on MXNet please share what >>>> you're working on? >>>> >>>> I'm working on >>>> -Redesigning the website >>>> <https://mxnet.incubator.apache.org/versions/master/index.html>. >>>> -Setting up a forum for user support. >>>> >>>> Seb Kiureghian >>>> >>> >> >> >> >> -- >> Sandeep Krishnamurthy > -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: Java API for MXNet
Agree with Sandeep, while I guess the performance won't change. But yes, benchmark talks. Moreover, in Scala package we use macros to generate operators automatically, which will require more efforts if we switch to pure Java. 2017-08-17 2:12 GMT+08:00 sandeep krishnamurthy : > The fastest way to get Java binding is through building Java native > wrappers on Scala package. > Disadvantages would be: >* *Bloated library size: *May not be suitable for users planning to use > Java APIs in Android of such smaller systems. >* *Performance:* Performance may not be as good as building directly > over JNI and implementing ground up. For example, taking NDArray dimensions > as Java ArrayList then converting it to Scala Seq to adapt for Scala > NDArray API and more such adapters. > > However, building ground up from JNI would be a huge effort without > actually getting feedback from users early. > > *My Plan:* > 1. Build Java interface on top of Scala package. > 2. Get early feedback from users. It may turn out Java is not a great > candidate for DL training jobs. > 3. Solidify the interface (APIs) for Java users. > 4. Do performance benchmarks to see Scala Native / Java interface. This > gives us comparable numbers on performance in Java. > 5. Over a period of time replace underlying Scala usage with JNI base and > native Java implementation. Provided feedback from users is positive. > > Comments/Suggestion? > > Regards, > Sandeep > > > On Wed, Aug 16, 2017 at 10:56 AM, YiZhi Liu wrote: > >> What Nan and I worried about is the re-implementation of something >> like https://github.com/apache/incubator-mxnet/blob/master/ >> scala-package/core/src/main/scala/ml/dmlc/mxnet/Model.scala#L246, >> and the executorManager, NDArray, KVStore ... it uses. >> >> the C API stays at the very low level. If this is the purpose, we can >> simply move ml.dmlc.mxnet.LibInfo to 'java' folder and compile without >> scala, no need to introduce JavaCPP. But I don't think this is what >> users want. >> >> 2017-08-17 1:41 GMT+08:00 Joern Kottmann : >> > There will be a new scala version one day, and the story we had with >> > going from 2.10 to 2.11 might just repeat. In the end if you make a >> > dependency using scala you just end up making it for the currently >> > popular scala versions. And that might be ok for projects with >> > developers who are familiar with these issues, but it is not ok for >> > java projects, where people might not expect it or know about these >> > problems. It just makes it harder to use. >> > >> > To me it looks like that the C API is very stable and used by all/most >> > other APIs. If we have a Java API - accessing the C API via JavaCPP - >> > then we should end up with a pretty stable solution and a lot the code >> > that is duplicated with the Scala API is the generated code. >> > >> > I think we should explore this possible way of implementing it with a >> > proof-of-concept. >> > >> > And if we have a well made Java API it might be something which maybe >> > wouldn't need a lot of additions to be pleasurable to use from scala. >> > >> > Jörn >> > >> > On Wed, Aug 16, 2017 at 6:45 PM, Nan Zhu wrote: >> >> I don't think there will be problems under "11", did the user see >> concrete >> >> errors? >> >> >> >> Best, >> >> >> >> Nan >> >> >> >> >> >> >> >> On Wed, Aug 16, 2017 at 9:30 AM, YiZhi Liu wrote: >> >> >> >>> Hi Nan, >> >>> >> >>> Users have 2.11, but with a different minor version, will it cause >> >>> conflicts? >> >>> >> >>> 2017-08-17 0:19 GMT+08:00 Nan Zhu : >> >>> > Hi, Yizhi, >> >>> > >> >>> > You mean users have 2.10 env while we assemble 2.11 in it? >> >>> > >> >>> > Best, >> >>> > >> >>> > Nan >> >>> > >> >>> > On Wed, Aug 16, 2017 at 9:08 AM, YiZhi Liu >> wrote: >> >>> > >> >>> >> Hi Joern, >> >>> >> >> >>> >> The point is that, the front is not a simple wrapper of c_api.h, as >> >>> >> you mentioned, which can be easily achieved by JavaCPP. >> >>> >> >> >>> >> I have noticed the potential conflicts between the assembled scala >> >>> &g
Re: Java API for MXNet
What Nan and I worried about is the re-implementation of something like https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/ml/dmlc/mxnet/Model.scala#L246, and the executorManager, NDArray, KVStore ... it uses. the C API stays at the very low level. If this is the purpose, we can simply move ml.dmlc.mxnet.LibInfo to 'java' folder and compile without scala, no need to introduce JavaCPP. But I don't think this is what users want. 2017-08-17 1:41 GMT+08:00 Joern Kottmann : > There will be a new scala version one day, and the story we had with > going from 2.10 to 2.11 might just repeat. In the end if you make a > dependency using scala you just end up making it for the currently > popular scala versions. And that might be ok for projects with > developers who are familiar with these issues, but it is not ok for > java projects, where people might not expect it or know about these > problems. It just makes it harder to use. > > To me it looks like that the C API is very stable and used by all/most > other APIs. If we have a Java API - accessing the C API via JavaCPP - > then we should end up with a pretty stable solution and a lot the code > that is duplicated with the Scala API is the generated code. > > I think we should explore this possible way of implementing it with a > proof-of-concept. > > And if we have a well made Java API it might be something which maybe > wouldn't need a lot of additions to be pleasurable to use from scala. > > Jörn > > On Wed, Aug 16, 2017 at 6:45 PM, Nan Zhu wrote: >> I don't think there will be problems under "11", did the user see concrete >> errors? >> >> Best, >> >> Nan >> >> >> >> On Wed, Aug 16, 2017 at 9:30 AM, YiZhi Liu wrote: >> >>> Hi Nan, >>> >>> Users have 2.11, but with a different minor version, will it cause >>> conflicts? >>> >>> 2017-08-17 0:19 GMT+08:00 Nan Zhu : >>> > Hi, Yizhi, >>> > >>> > You mean users have 2.10 env while we assemble 2.11 in it? >>> > >>> > Best, >>> > >>> > Nan >>> > >>> > On Wed, Aug 16, 2017 at 9:08 AM, YiZhi Liu wrote: >>> > >>> >> Hi Joern, >>> >> >>> >> The point is that, the front is not a simple wrapper of c_api.h, as >>> >> you mentioned, which can be easily achieved by JavaCPP. >>> >> >>> >> I have noticed the potential conflicts between the assembled scala >>> >> library and the one in users' environment. Can we remove the scala >>> >> library from the assembly jar? @Nan It wouldn't be a problem since the >>> >> scala libraries with same major version are compatible. >>> >> >>> >> 2017-08-16 23:49 GMT+08:00 Joern Kottmann : >>> >> > Hello, >>> >> > >>> >> > I personally had quite some issues with Scala dependencies in >>> >> > different versions and Spark, where one version is not compatible with >>> >> > the other version. Then you need to debug the dependency tree to find >>> >> > the places where the versions don't match. Every project which would >>> >> > like to use MXnet then has to depend on Scala and might also get >>> >> > conflicts if other dependencies depend on different Scala versions. >>> >> > Probably something which will cause issues for some of your users. >>> >> > Users who want to use Java might not be familiar with Scala dependency >>> >> > problems and have a hard time resolving them by getting strange error >>> >> > messages. >>> >> > >>> >> > The JNI layer could be generated with JavaCPP, then we would not need >>> >> > to write/maintain the C and the jvm side for that our self. >>> >> > A good example of JavaCPP and Scala usage is Apache Mahout [1]. >>> >> > >>> >> > Even if we don't use JavaCPP, the JNI layer should be easy to get into >>> >> > a state where both can share it, the current Scala JNI layers LibInfo >>> >> > classes could be converted to Java classes and would in most cases >>> >> > require only minor changes in the Scala code. >>> >> > >>> >> > Jörn >>> >> > >>> >> > [1] https://github.com/apache/mahout/tree/master/viennacl/src/main >>> >> > >>> >> >
Re: Java API for MXNet
Hi Nan, Users have 2.11, but with a different minor version, will it cause conflicts? 2017-08-17 0:19 GMT+08:00 Nan Zhu : > Hi, Yizhi, > > You mean users have 2.10 env while we assemble 2.11 in it? > > Best, > > Nan > > On Wed, Aug 16, 2017 at 9:08 AM, YiZhi Liu wrote: > >> Hi Joern, >> >> The point is that, the front is not a simple wrapper of c_api.h, as >> you mentioned, which can be easily achieved by JavaCPP. >> >> I have noticed the potential conflicts between the assembled scala >> library and the one in users' environment. Can we remove the scala >> library from the assembly jar? @Nan It wouldn't be a problem since the >> scala libraries with same major version are compatible. >> >> 2017-08-16 23:49 GMT+08:00 Joern Kottmann : >> > Hello, >> > >> > I personally had quite some issues with Scala dependencies in >> > different versions and Spark, where one version is not compatible with >> > the other version. Then you need to debug the dependency tree to find >> > the places where the versions don't match. Every project which would >> > like to use MXnet then has to depend on Scala and might also get >> > conflicts if other dependencies depend on different Scala versions. >> > Probably something which will cause issues for some of your users. >> > Users who want to use Java might not be familiar with Scala dependency >> > problems and have a hard time resolving them by getting strange error >> > messages. >> > >> > The JNI layer could be generated with JavaCPP, then we would not need >> > to write/maintain the C and the jvm side for that our self. >> > A good example of JavaCPP and Scala usage is Apache Mahout [1]. >> > >> > Even if we don't use JavaCPP, the JNI layer should be easy to get into >> > a state where both can share it, the current Scala JNI layers LibInfo >> > classes could be converted to Java classes and would in most cases >> > require only minor changes in the Scala code. >> > >> > Jörn >> > >> > [1] https://github.com/apache/mahout/tree/master/viennacl/src/main >> > >> > On Wed, Aug 16, 2017 at 5:30 PM, Nan Zhu wrote: >> >> I agree with Yizhi >> >> >> >> My major concern is the duplicate implementations, which are usually >> one of >> >> the major sources of bugs, especially with two languages which are >> >> naturally interactive (OK, Calling Scala from Java might need some more >> >> efforts). It is just like we provide C++ & C APIs of MxNet in two >> separated >> >> packages. >> >> >> >> About dependency problem, when you say "As far as I see this has the >> great >> >> disadvantage that the Java API would force Scala as a dependency onto >> the >> >> java users.", would you please give a concrete example causing critical >> >> issues? >> >> >> >> Best, >> >> >> >> Nan >> >> >> >> >> >> >> >> On Wed, Aug 16, 2017 at 8:19 AM, YiZhi Liu wrote: >> >> >> >>> Hi, >> >>> >> >>> If we build the Java API from the very beginning, i.e. the JNI part, >> >>> we have to rewrite the codes for training, predict, inferShape, etc. >> >>> It would be too heavy to maintain a totally new front language. >> >>> >> >>> As far as I see, I don't think Scala library dependency would be a big >> >>> problem in most cases, unless we are going to use it in embedded >> >>> devices. Could you illustrate some use-cases where you cannot involve >> >>> Scala dependencies? >> >>> >> >>> 2017-08-16 22:13 GMT+08:00 Joern Kottmann : >> >>> > Hello, >> >>> > >> >>> > the approach which is taken by Spark is described here [1]. >> >>> > >> >>> > As far as I see this has the great disadvantage that the Java API >> >>> > would force Scala as a dependency onto the java users. >> >>> > For a library it is always a great advantage if it doesn't have many >> >>> > dependencies, or zero dependencies. In our case it could be quite >> >>> > realistic to have a thin wrapper around the C API without needing any >> >>> > other dependencies (or only dependencies which can't be avoided). >> >>> > >&
Re: Java API for MXNet
Hi Joern, The point is that, the front is not a simple wrapper of c_api.h, as you mentioned, which can be easily achieved by JavaCPP. I have noticed the potential conflicts between the assembled scala library and the one in users' environment. Can we remove the scala library from the assembly jar? @Nan It wouldn't be a problem since the scala libraries with same major version are compatible. 2017-08-16 23:49 GMT+08:00 Joern Kottmann : > Hello, > > I personally had quite some issues with Scala dependencies in > different versions and Spark, where one version is not compatible with > the other version. Then you need to debug the dependency tree to find > the places where the versions don't match. Every project which would > like to use MXnet then has to depend on Scala and might also get > conflicts if other dependencies depend on different Scala versions. > Probably something which will cause issues for some of your users. > Users who want to use Java might not be familiar with Scala dependency > problems and have a hard time resolving them by getting strange error > messages. > > The JNI layer could be generated with JavaCPP, then we would not need > to write/maintain the C and the jvm side for that our self. > A good example of JavaCPP and Scala usage is Apache Mahout [1]. > > Even if we don't use JavaCPP, the JNI layer should be easy to get into > a state where both can share it, the current Scala JNI layers LibInfo > classes could be converted to Java classes and would in most cases > require only minor changes in the Scala code. > > Jörn > > [1] https://github.com/apache/mahout/tree/master/viennacl/src/main > > On Wed, Aug 16, 2017 at 5:30 PM, Nan Zhu wrote: >> I agree with Yizhi >> >> My major concern is the duplicate implementations, which are usually one of >> the major sources of bugs, especially with two languages which are >> naturally interactive (OK, Calling Scala from Java might need some more >> efforts). It is just like we provide C++ & C APIs of MxNet in two separated >> packages. >> >> About dependency problem, when you say "As far as I see this has the great >> disadvantage that the Java API would force Scala as a dependency onto the >> java users.", would you please give a concrete example causing critical >> issues? >> >> Best, >> >> Nan >> >> >> >> On Wed, Aug 16, 2017 at 8:19 AM, YiZhi Liu wrote: >> >>> Hi, >>> >>> If we build the Java API from the very beginning, i.e. the JNI part, >>> we have to rewrite the codes for training, predict, inferShape, etc. >>> It would be too heavy to maintain a totally new front language. >>> >>> As far as I see, I don't think Scala library dependency would be a big >>> problem in most cases, unless we are going to use it in embedded >>> devices. Could you illustrate some use-cases where you cannot involve >>> Scala dependencies? >>> >>> 2017-08-16 22:13 GMT+08:00 Joern Kottmann : >>> > Hello, >>> > >>> > the approach which is taken by Spark is described here [1]. >>> > >>> > As far as I see this has the great disadvantage that the Java API >>> > would force Scala as a dependency onto the java users. >>> > For a library it is always a great advantage if it doesn't have many >>> > dependencies, or zero dependencies. In our case it could be quite >>> > realistic to have a thin wrapper around the C API without needing any >>> > other dependencies (or only dependencies which can't be avoided). >>> > >>> > The JNI layer could easily be shared between the Java and Scala API. >>> > As far as I understand is the JNI layer in the Scala API anyway >>> > private and a change to it wouldn't require that the public part of >>> > the Scala API is changed. >>> > >>> > What do you think? >>> > >>> > Jörn >>> > >>> > [1] https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals >>> > >>> > On Wed, Aug 16, 2017 at 3:39 PM, YiZhi Liu wrote: >>> >> Hi Joern, >>> >> >>> >> I suggest to build Java API as a wrapper of Scala API, re-use most of >>> >> the procedures. Referring to the Java API in Apache Spark. >>> >> >>> >> 2017-08-16 18:21 GMT+08:00 Joern Kottmann : >>> >>> Hello all, >>> >>> >>> >>> I would like to propose the addition of a Java API to MXNet. >>> >>&
Re: Java API for MXNet
Hi, If we build the Java API from the very beginning, i.e. the JNI part, we have to rewrite the codes for training, predict, inferShape, etc. It would be too heavy to maintain a totally new front language. As far as I see, I don't think Scala library dependency would be a big problem in most cases, unless we are going to use it in embedded devices. Could you illustrate some use-cases where you cannot involve Scala dependencies? 2017-08-16 22:13 GMT+08:00 Joern Kottmann : > Hello, > > the approach which is taken by Spark is described here [1]. > > As far as I see this has the great disadvantage that the Java API > would force Scala as a dependency onto the java users. > For a library it is always a great advantage if it doesn't have many > dependencies, or zero dependencies. In our case it could be quite > realistic to have a thin wrapper around the C API without needing any > other dependencies (or only dependencies which can't be avoided). > > The JNI layer could easily be shared between the Java and Scala API. > As far as I understand is the JNI layer in the Scala API anyway > private and a change to it wouldn't require that the public part of > the Scala API is changed. > > What do you think? > > Jörn > > [1] https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals > > On Wed, Aug 16, 2017 at 3:39 PM, YiZhi Liu wrote: >> Hi Joern, >> >> I suggest to build Java API as a wrapper of Scala API, re-use most of >> the procedures. Referring to the Java API in Apache Spark. >> >> 2017-08-16 18:21 GMT+08:00 Joern Kottmann : >>> Hello all, >>> >>> I would like to propose the addition of a Java API to MXNet. >>> >>> There has been some previous work done for the Scala API, and it makes >>> sense to at least share the JNI layer between the two. >>> >>> The Java API probably should be aligned with the Python API (and >>> others which exist already) with a few changes to give it a native >>> Java feel. >>> >>> As far as I understand there are multiple people interested to work on >>> this and it would be good to maybe come up with a written proposal on >>> how things should be. >>> >>> My motivation is to get a Java API which can be used by Apache OpenNLP >>> to solve various NLP tasks using Deep Learning based approaches and I >>> am also interested to work on MXNet. >>> >>> Jörn >> >> >> >> -- >> Yizhi Liu >> DMLC member >> Technical Manager >> Qihoo 360 Inc, Shanghai, China -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: Java API for MXNet
Hi Joern, I suggest to build Java API as a wrapper of Scala API, re-use most of the procedures. Referring to the Java API in Apache Spark. 2017-08-16 18:21 GMT+08:00 Joern Kottmann : > Hello all, > > I would like to propose the addition of a Java API to MXNet. > > There has been some previous work done for the Scala API, and it makes > sense to at least share the JNI layer between the two. > > The Java API probably should be aligned with the Python API (and > others which exist already) with a few changes to give it a native > Java feel. > > As far as I understand there are multiple people interested to work on > this and it would be good to maybe come up with a written proposal on > how things should be. > > My motivation is to get a Java API which can be used by Apache OpenNLP > to solve various NLP tasks using Deep Learning based approaches and I > am also interested to work on MXNet. > > Jörn -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: IMPORTANT: Link accounts to Apache today 7/12
Done. Apache User ID: liuyizhi 2017-07-13 17:53 GMT+08:00 Chiyuan Zhang : > apache id is: pluskid > Thanks! > > chiyuan > > On Wed, Jul 12, 2017 at 11:02 PM, Ly Nguyen wrote: >> >> As mentioned in the migration FAQ, if you currently have any kind of write >> access to MXNet repo, once migration happens, YOU WILL LOSE IT unless you >> do the following. >> >> If you haven't already, please link your Apache accounts (you need to >> already be a committer) to Gitbox here: https://gitbox.apache.org/setup/ >> You will need to enable 2 Factor Auth and make sure all three boxes on the >> page show check marks. >> >> Please respond here with your Apache user id once complete. >> >> @pono or @henri will have to make sure that committers and PMC members are >> added to the `mxnet` group https://whimsy.apache.org/roster/ppmc/mxnet > > -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
Re: New Logo?
+1 2017-02-22 9:08 GMT+08:00 Ziheng Jiang : > +1 > > On Tue, Feb 21, 2017 at 4:58 PM Yutian Li wrote: > >> +1 >> >> On Tue, Feb 21, 2017 at 08:26:44AM -0800, Nan Zhu wrote: >> > +1 >> > >> > On Mon, Feb 20, 2017 at 11:57 PM, Henri Yandell >> wrote: >> > >> > > We should do a vote on this, now that we have everyone here. >> > > >> > > Voting is described here: http://apache.org/foundation/voting.html >> > > >> > > (though basically it comes down to an email saying: >> > > >> > > >> > > Let's vote on a thingy >> > > [ ] +1; Yay, thingy >> > > [ ] -1; Hate it, because >> > > >> > > Making sure that enough time elapses (minimum 3 days), and that the >> project >> > > has voted (minimum 3 PMC members); and then getting the Incubator PMC >> to >> > > sign off as we operate as a child of theirs. >> > > >> > > >> > > If anyone wants to include details and propose a vote, that would be >> > > brilliant :) I'm happy to review an email if anyone is unsure and >> wants to >> > > send a draft my way. >> > > >> > > Hen >> > > >> > > >> > > On Thu, Jan 26, 2017 at 2:55 PM, Spisak, Joseph >> > > wrote: >> > > >> > > > Thanks Henri, we’ve done some votes in the past on GitHub but haven’t >> > > > really concluded. It would be great if the community came up with >> some >> > > > proposals that we could vote on. The past vote can be found here: >> > > > https://github.com/dmlc/mxnet/issues/4392 >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > Cheers, >> > > > >> > > > -Joe >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *From: *Henri Yandell >> > > > *Date: *Thursday, January 26, 2017 at 2:47 PM >> > > > *To: *"dev@mxnet.incubator.apache.org" < >> dev@mxnet.incubator.apache.org> >> > > > *Cc: *"Spisak, Joseph" >> > > > *Subject: *New Logo? >> > > > >> > > > >> > > > >> > > > Joe - from a private conversation I believe you wanted to >> propose/offer a >> > > > new logo for the project? >> > > > >> > > > Aka *nudge* to bring the topic up here as a first item of business :) >> > > > >> > > > Hen >> > > > >> > > >> > -- > Ziheng Jiang > Fudan University | Computer Science -- Yizhi Liu DMLC member Technical Manager Qihoo 360 Inc, Shanghai, China
