Re: Regressions in NDArrayIter
I am looking into it. Could you provide the shape of NDArrayIter and minimal reproducible code? Thank you so much! Alfredo Luque 於 2018年9月11日 週二 下午6:05寫道： > Looks like https://github.com/apache/incubator-mxnet/pull/12285 broke a > ton > of our test cases iterating over 3D NDArray instances (eg; MNIST) by > creating an index out of range. > > Stacktrace: > > .com/airbnb/bighead/python/bighead/ml_frameworks/mxnet/gluon.py", line > 434, in transform > for batch in data_iter: > File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", > line 228, in __next__ > return self.next() > File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", > line 680, in next > label = self.getlabel() > File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", > line 750, in getlabel > return self._batchify(self.label) > File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", > line 732, in _batchify > first_data = self._getdata(data_source, start=self.cursor) > File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", > line 694, in _getdata > end = end if end is not None else data_source[0][1].shape[0] > IndexError: list index out of range > > I’ve created an issue at > https://github.com/apache/incubator-mxnet/issues/12526 > > > We’ll be pinning to the previous build until it’s reverted/patched, but let > us know if we can help provide more regression tests here. > > — > Alfredo Luque > Software Engineer > Machine Learning Infrastructure > Airbnb > San Francisco, CA >
Re: Off-Heap Memory Management in MXNet Scala
do you log on finalize() if the object wasn’t properly freed (ie NDArray.finalize())? is that available in Scala? On Tue, Sep 11, 2018 at 6:12 PM Qing Lan wrote: > Nice document! Way better than current .dispose() in Scala! > > Thanks, > Qing > > On 9/11/18, 6:04 PM, "Chris Olivier" wrote: > > wow, incredible document! > > On Tue, Sep 11, 2018 at 2:37 PM Naveen Swamy > wrote: > > > Hi All, > > > > I am working on managing Off-Heap Memory Management and have written > a > > proposal here based on my prototype and research I did. > > > > Please review the doc and provide your feedback ? > > > > > https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management > > > > I had offline discussion with a few people I work with and added > their > > feedback to the doc as well. > > > > Thanks, Naveen > > > > >
Re: Off-Heap Memory Management in MXNet Scala
Nice document! Way better than current .dispose() in Scala! Thanks, Qing On 9/11/18, 6:04 PM, "Chris Olivier" wrote: wow, incredible document! On Tue, Sep 11, 2018 at 2:37 PM Naveen Swamy wrote: > Hi All, > > I am working on managing Off-Heap Memory Management and have written a > proposal here based on my prototype and research I did. > > Please review the doc and provide your feedback ? > > https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management > > I had offline discussion with a few people I work with and added their > feedback to the doc as well. > > Thanks, Naveen >
Re: Off-Heap Memory Management in MXNet Scala
wow, incredible document! On Tue, Sep 11, 2018 at 2:37 PM Naveen Swamy wrote: > Hi All, > > I am working on managing Off-Heap Memory Management and have written a > proposal here based on my prototype and research I did. > > Please review the doc and provide your feedback ? > > https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management > > I had offline discussion with a few people I work with and added their > feedback to the doc as well. > > Thanks, Naveen >
Regressions in NDArrayIter
Looks like https://github.com/apache/incubator-mxnet/pull/12285 broke a ton of our test cases iterating over 3D NDArray instances (eg; MNIST) by creating an index out of range. Stacktrace: .com/airbnb/bighead/python/bighead/ml_frameworks/mxnet/gluon.py", line 434, in transform for batch in data_iter: File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", line 228, in __next__ return self.next() File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", line 680, in next label = self.getlabel() File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", line 750, in getlabel return self._batchify(self.label) File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", line 732, in _batchify first_data = self._getdata(data_source, start=self.cursor) File "/anaconda3/envs/py36/lib/python3.6/site-packages/mxnet/io/io.py", line 694, in _getdata end = end if end is not None else data_source[0][1].shape[0] IndexError: list index out of range I’ve created an issue at https://github.com/apache/incubator-mxnet/issues/12526 We’ll be pinning to the previous build until it’s reverted/patched, but let us know if we can help provide more regression tests here. — Alfredo Luque Software Engineer Machine Learning Infrastructure Airbnb San Francisco, CA
Re: Error Publishing OSX package to Maven
Qing helped test by excluding the bundle/source. It still creates 2 repos in Staging. I am not sure if Maven Version or some other maven-plugin difference is causing this issue, there was no issue publishing to Linux-CPU and Linux-GPU from the same code. For now, I have manually uploaded the artifacts(generated through maven release prepare/perform) to the Staging repo so we can make it available for OSX users. This issue needs to be investigated independently. -Naveen On Tue, Sep 11, 2018 at 2:03 PM, Naveen Swamy wrote: > that seems reasonable. I also asked this question on d...@maven.apache.org > to see if there is anything obvious that we missed. > > https://lists.apache.org/thread.html/def6e5c6c47ab2f39592a1fe060b6c > fd0008d303a2b5c814545d231b@%3Cdev.maven.apache.org%3E > > On Tue, Sep 11, 2018 at 11:25 AM, Carin Meier > wrote: > >> I would suggest trying >> >> * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. >> and seeing if that solves the problem and allows publishing to staging. >> >> If it does work then it helps to identity the cause. >> If the effort does not take to long. I would recommend timeboxing the >> effort and if it is going to take more effort to investigate, we can go >> with announcing later for OSX. >> >> >> - Carin >> >> On Tue, Sep 11, 2018 at 2:10 PM Naveen Swamy wrote: >> >> > hey all, >> > >> > I am working on publishing the 1.3.0 Scala package to Maven and >> > encountering a error when I am about to Close the Repo on Nexus. When I >> > publish the OSX package to Staging, the artifacts gets split into 2 >> > repositories and when I close[1] the repo(to make it available for use) >> it >> > fails since the signature files are in a different repo. This is how the >> > artifacts are getting split >> > >> > Repo1: orgapachemxnet-1018 >> > === >> > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.md5 >> > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.sha1 >> > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.asc >> > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.md5 >> > mxnet-full_2.11-osx-x86_64-cpu-src.jar.md5 >> > mxnet-full_2.11-osx-x86_64-cpu-src.jar.sha1 >> > mxnet-full_2.11-osx-x86_64-cpu.jar >> > mxnet-full_2.11-osx-x86_64-cpu.jar.asc >> > mxnet-full_2.11-osx-x86_64-cpu.pom.md5 >> > mxnet-full_2.11-osx-x86_64-cpu.pom.sha1 >> > >> > >> > Repo2: orgapachemxnet-1019 >> > === >> > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar >> > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.asc >> > mxnet-full_2.11-osx-x86_64-cpu-sources.jar >> > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.sha1 >> > mxnet-full_2.11-osx-x86_64-cpu-src.jar >> > mxnet-full_2.11-osx-x86_64-cpu-src.jar.asc >> > mxnet-full_2.11-osx-x86_64-cpu-jar.md5 >> > mxnet-full_2.11-osx-x86_64-cpu.jar.sha1 >> > mxnet-full_2.11-osx-x86_64-cpu.pom >> > mxnet-full_2.11-osx-x86_64-cpu.pom.asc >> > >> > This was not an issue with Linux-CPU and Linux-GPU versions for 1.3.0 >> and >> > suspicion is the new Source and documentation jar files being added as a >> > part of 1.3.0 for OSX. >> > >> > There are 2 options >> > * Continue to investigate the issue and announce later for OSX (after >> the >> > general announcement) >> > * Remove bundle/source jar from 1.3.0 and attempt publishing OSX >> package. >> > >> > What do you guys suggest? >> > >> > >> > 1) Closing Staged Repo - >> > >> > https://central.sonatype.org/pages/releasing-the-deployment. >> html#locate-and-examine-your-staging-repository >> > 2) Scala Release Process: >> > >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet- >> Scala+Release+Process >> > >> > >
Off-Heap Memory Management in MXNet Scala
Hi All, I am working on managing Off-Heap Memory Management and have written a proposal here based on my prototype and research I did. Please review the doc and provide your feedback ? https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management I had offline discussion with a few people I work with and added their feedback to the doc as well. Thanks, Naveen
Re: Error Publishing OSX package to Maven
that seems reasonable. I also asked this question on d...@maven.apache.org to see if there is anything obvious that we missed. https://lists.apache.org/thread.html/def6e5c6c47ab2f39592a1fe060b6cfd0008d303a2b5c814545d231b@%3Cdev.maven.apache.org%3E On Tue, Sep 11, 2018 at 11:25 AM, Carin Meier wrote: > I would suggest trying > > * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. > and seeing if that solves the problem and allows publishing to staging. > > If it does work then it helps to identity the cause. > If the effort does not take to long. I would recommend timeboxing the > effort and if it is going to take more effort to investigate, we can go > with announcing later for OSX. > > > - Carin > > On Tue, Sep 11, 2018 at 2:10 PM Naveen Swamy wrote: > > > hey all, > > > > I am working on publishing the 1.3.0 Scala package to Maven and > > encountering a error when I am about to Close the Repo on Nexus. When I > > publish the OSX package to Staging, the artifacts gets split into 2 > > repositories and when I close[1] the repo(to make it available for use) > it > > fails since the signature files are in a different repo. This is how the > > artifacts are getting split > > > > Repo1: orgapachemxnet-1018 > > === > > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.md5 > > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.sha1 > > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.asc > > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.md5 > > mxnet-full_2.11-osx-x86_64-cpu-src.jar.md5 > > mxnet-full_2.11-osx-x86_64-cpu-src.jar.sha1 > > mxnet-full_2.11-osx-x86_64-cpu.jar > > mxnet-full_2.11-osx-x86_64-cpu.jar.asc > > mxnet-full_2.11-osx-x86_64-cpu.pom.md5 > > mxnet-full_2.11-osx-x86_64-cpu.pom.sha1 > > > > > > Repo2: orgapachemxnet-1019 > > === > > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar > > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.asc > > mxnet-full_2.11-osx-x86_64-cpu-sources.jar > > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.sha1 > > mxnet-full_2.11-osx-x86_64-cpu-src.jar > > mxnet-full_2.11-osx-x86_64-cpu-src.jar.asc > > mxnet-full_2.11-osx-x86_64-cpu-jar.md5 > > mxnet-full_2.11-osx-x86_64-cpu.jar.sha1 > > mxnet-full_2.11-osx-x86_64-cpu.pom > > mxnet-full_2.11-osx-x86_64-cpu.pom.asc > > > > This was not an issue with Linux-CPU and Linux-GPU versions for 1.3.0 and > > suspicion is the new Source and documentation jar files being added as a > > part of 1.3.0 for OSX. > > > > There are 2 options > > * Continue to investigate the issue and announce later for OSX (after the > > general announcement) > > * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. > > > > What do you guys suggest? > > > > > > 1) Closing Staged Repo - > > > > https://central.sonatype.org/pages/releasing-the- > deployment.html#locate-and-examine-your-staging-repository > > 2) Scala Release Process: > > > > https://cwiki.apache.org/confluence/display/MXNET/ > MXNet-Scala+Release+Process > > >
Re: Error Publishing OSX package to Maven
I would suggest trying * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. and seeing if that solves the problem and allows publishing to staging. If it does work then it helps to identity the cause. If the effort does not take to long. I would recommend timeboxing the effort and if it is going to take more effort to investigate, we can go with announcing later for OSX. - Carin On Tue, Sep 11, 2018 at 2:10 PM Naveen Swamy wrote: > hey all, > > I am working on publishing the 1.3.0 Scala package to Maven and > encountering a error when I am about to Close the Repo on Nexus. When I > publish the OSX package to Staging, the artifacts gets split into 2 > repositories and when I close[1] the repo(to make it available for use) it > fails since the signature files are in a different repo. This is how the > artifacts are getting split > > Repo1: orgapachemxnet-1018 > === > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.md5 > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.sha1 > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.asc > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.md5 > mxnet-full_2.11-osx-x86_64-cpu-src.jar.md5 > mxnet-full_2.11-osx-x86_64-cpu-src.jar.sha1 > mxnet-full_2.11-osx-x86_64-cpu.jar > mxnet-full_2.11-osx-x86_64-cpu.jar.asc > mxnet-full_2.11-osx-x86_64-cpu.pom.md5 > mxnet-full_2.11-osx-x86_64-cpu.pom.sha1 > > > Repo2: orgapachemxnet-1019 > === > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar > mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.asc > mxnet-full_2.11-osx-x86_64-cpu-sources.jar > mxnet-full_2.11-osx-x86_64-cpu-sources.jar.sha1 > mxnet-full_2.11-osx-x86_64-cpu-src.jar > mxnet-full_2.11-osx-x86_64-cpu-src.jar.asc > mxnet-full_2.11-osx-x86_64-cpu-jar.md5 > mxnet-full_2.11-osx-x86_64-cpu.jar.sha1 > mxnet-full_2.11-osx-x86_64-cpu.pom > mxnet-full_2.11-osx-x86_64-cpu.pom.asc > > This was not an issue with Linux-CPU and Linux-GPU versions for 1.3.0 and > suspicion is the new Source and documentation jar files being added as a > part of 1.3.0 for OSX. > > There are 2 options > * Continue to investigate the issue and announce later for OSX (after the > general announcement) > * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. > > What do you guys suggest? > > > 1) Closing Staged Repo - > > https://central.sonatype.org/pages/releasing-the-deployment.html#locate-and-examine-your-staging-repository > 2) Scala Release Process: > > https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process >
Error Publishing OSX package to Maven
hey all, I am working on publishing the 1.3.0 Scala package to Maven and encountering a error when I am about to Close the Repo on Nexus. When I publish the OSX package to Staging, the artifacts gets split into 2 repositories and when I close[1] the repo(to make it available for use) it fails since the signature files are in a different repo. This is how the artifacts are getting split Repo1: orgapachemxnet-1018 === mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.md5 mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.sha1 mxnet-full_2.11-osx-x86_64-cpu-sources.jar.asc mxnet-full_2.11-osx-x86_64-cpu-sources.jar.md5 mxnet-full_2.11-osx-x86_64-cpu-src.jar.md5 mxnet-full_2.11-osx-x86_64-cpu-src.jar.sha1 mxnet-full_2.11-osx-x86_64-cpu.jar mxnet-full_2.11-osx-x86_64-cpu.jar.asc mxnet-full_2.11-osx-x86_64-cpu.pom.md5 mxnet-full_2.11-osx-x86_64-cpu.pom.sha1 Repo2: orgapachemxnet-1019 === mxnet-full_2.11-osx-x86_64-cpu-bundle.jar mxnet-full_2.11-osx-x86_64-cpu-bundle.jar.asc mxnet-full_2.11-osx-x86_64-cpu-sources.jar mxnet-full_2.11-osx-x86_64-cpu-sources.jar.sha1 mxnet-full_2.11-osx-x86_64-cpu-src.jar mxnet-full_2.11-osx-x86_64-cpu-src.jar.asc mxnet-full_2.11-osx-x86_64-cpu-jar.md5 mxnet-full_2.11-osx-x86_64-cpu.jar.sha1 mxnet-full_2.11-osx-x86_64-cpu.pom mxnet-full_2.11-osx-x86_64-cpu.pom.asc This was not an issue with Linux-CPU and Linux-GPU versions for 1.3.0 and suspicion is the new Source and documentation jar files being added as a part of 1.3.0 for OSX. There are 2 options * Continue to investigate the issue and announce later for OSX (after the general announcement) * Remove bundle/source jar from 1.3.0 and attempt publishing OSX package. What do you guys suggest? 1) Closing Staged Repo - https://central.sonatype.org/pages/releasing-the-deployment.html#locate-and-examine-your-staging-repository 2) Scala Release Process: https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process
Re: Enable Travis CI for Apache MXNet
Done, here is the ticket: https://issues.apache.org/jira/browse/INFRA-17004 On 10.09.2018 14:49, Marco de Abreu wrote: Hello Sebastian, could you please file a ticket with Apache Infra to enable Travis CI for our main repository? The discussion thread is available at https://lists.apache.org/thread.html/3871eb5a7bc31eb467f68935f4a56469292b03a33f1693cdb9a98f31@%3Cdev.mxnet.apache.org%3E . The settings should be branch- as well as PR-validation. The reported status should be set to "not required", thus not blocking any PRs if that validation pipeline fails. Best regards, Marco
GitHub having issues today
FYI GitHub is having issues today. It's severe enough that it's blocking my ability to validate PRs. I'm sure it will be resolved soon, but it might be a good day to read a few papers ;-). https://status.github.com/messages -Kellen
