Progress since the last update: 1. 2 more fixes [1] [2] were picked to v1.5.x; 2. 1 cherry pick PR [3] was opened for TensorRT; 3. The nightly build issue [4] was removed from the scope of 1.5.1; 4. GPU OOM issue [5] was removed from the scope of 1.5.1. 5. So far, nightly test shows the release branch is healthy.
Opens: 1. [3] is stopped by the CI; 2. [6] is proposed but not picked to v1.5.x branch. @Aaron could you please update the status? Please find details on https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status Thanks, -tao [1] https://github.com/apache/incubator-mxnet/pull/16044 [2] https://github.com/apache/incubator-mxnet/pull/15803 [3] https://github.com/apache/incubator-mxnet/pull/16043 [4] https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-516937546 [5] https://github.com/apache/incubator-mxnet/issues/15703 [6] https://github.com/apache/incubator-mxnet/pull/15608 On Fri, Aug 30, 2019 at 7:16 AM Lai Wei <roywei...@gmail.com> wrote: > Hi Tao, > > Just checked 1.5.x nightly build is passing, so 10 is not needed. I moved > it so 1.6.0 scope. > > Thanks > > > Best Regards > > Lai > > > On Thu, Aug 29, 2019 at 8:12 AM Tao Lv <ta...@apache.org> wrote: > > > @Aaron, > > Thank you for looking into these two issues. I have removed the #15609 > from > > the scope of 1.5.1. Please let me know if you have any update about > #15608. > > > > @Lai, > > I'm fine with the decision. License issue about MKL-DNN, cub and pybind > is > > moved to next release. > > > > @Sam, > > I also removed the sidebar issue [3] from the scope of 1.5.1. Besides, I > > notice one of your cherry picks is stopped by the CI. Please take a look > at > > it. Thanks. > > > > *Nice progress since the last update:* > > 1. Per the discussion, we decided to remove #15609, the license issue > about > > MKL-DNN, cub and pybind, and the sidebar issue [3] from the scope of > 1.5.1 > > patch release; > > 2. 3 fixes [4] [5] [6] were merged into the v1.5.x branch. > > > > *Opens (suggested owners are highlighted):* > > 1. @Aaron is working on #15608 to see if we can have it in v1.5.x; > > 2. Two cherry pick PRs [7] [8] cannot pass the CI. I have pinged the > > authors to take a look at the CI failures. > > 3. @Kellen proposed 5 fixes [9] for TensorRT but till now only 3 are > picked > > to v1.5.x. Please help to confirm if the other 2 are still needed. > > 4. Sorry that I missed the proposal for fixing the nightly build [10] in > > previous update. @Lai, can you help to confirm if it's still valid? > > 5. @Lin please help to make a conclusion for the GPU OOM issue caused by > > topk regression [11]. If it cannot be addressed on v1.5.x branch, I will > > remove it from the scope of this release and mark it as a known issue in > > the release note. > > > > Please find the details in > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status > > . > > > > Thanks, > > -tao > > > > [1] https://github.com/apache/incubator-mxnet/pull/15609 > > [2] https://github.com/apache/incubator-mxnet/pull/15608 > > [3] https://github.com/apache/incubator-mxnet/issues/15200 > > [4] https://github.com/apache/incubator-mxnet/pull/16029 > > [5] https://github.com/apache/incubator-mxnet/pull/16026 > > [6] https://github.com/apache/incubator-mxnet/pull/16028 > > [7] https://github.com/apache/incubator-mxnet/pull/15803 > > [8] https://github.com/apache/incubator-mxnet/pull/16027 > > [9] > > > > > https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-520688668 > > [10] > > > > > https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-516937546 > > [11] https://github.com/apache/incubator-mxnet/issues/15703 > > > > > > > > On Thu, Aug 29, 2019 at 1:06 AM Skalicky, Sam <sska...@amazon.com.invalid > > > > wrote: > > > > > Hi Tao, > > > > > > I just talked with Aaron, lets leave the sidebar issue for later. > > > > > > I created PRs in the v1.5.x branch to cherry pick the fixes into the > > 1.5.1 > > > release: > > > https://github.com/apache/incubator-mxnet/pull/16027 > > > https://github.com/apache/incubator-mxnet/pull/16028 > > > > > > Thanks for your work on this release! > > > Sam > > > > > > On Aug 28, 2019, at 9:35 AM, Lai Wei <roywei...@gmail.com<mailto: > > > roywei...@gmail.com>> wrote: > > > > > > Hi, > > > > > > Regrading the license issue[1], we still have item 3, 4, 5 left. > > > I think it's better to remove them from 1.5.1 release scope and target > > for > > > 1.6.0 as it need more time and requires changes that should not go into > > > patch release. > > > > > > > > > [1] https://github.com/apache/incubator-mxnet/issues/15542 > > > > > > Best Regards > > > > > > Lai > > > > > > > > > On Wed, Aug 28, 2019 at 9:20 AM Aaron Markham < > aaron.s.mark...@gmail.com > > > <mailto:aaron.s.mark...@gmail.com>> > > > wrote: > > > > > > 5 no. Install page defaults to master so you don't need to pick it. > > > 6 probably, but there might be other PRs needed. I'd check out the > branch > > > and attempt the install across platforms to be sure. > > > > > > On Wed, Aug 28, 2019, 08:55 Tao Lv <ta...@apache.org<mailto: > > > ta...@apache.org>> wrote: > > > > > > Hi Aaron, > > > > > > They were proposed to be ported to v1.5.x at the beginning of the > > > discussion but I didn't see any action for that. So I'm wondering if > > > they're still needed. I asked for that in the last update on 8/20 but > > > didn't get a response. > > > > > > If they're still needed, I hope someone who is more familiar with Julia > > > frontend can help to cherry pick the commits to the v1.5.x branch. > > > > > > thanks, > > > -tao > > > > > > On Wed, Aug 28, 2019 at 11:43 PM Aaron Markham < > > > aaron.s.mark...@gmail.com<mailto:aaron.s.mark...@gmail.com>> > > > wrote: > > > > > > I don't see any request for action on the Julia PRs: 5 or 6. > > > We didn't put the change in right away because we wanted it to not > > > break > > > anything. But the changes are needed to make Julia setup more seamless. > > > > > > What "update" is needed? > > > > > > > > > On Wed, Aug 28, 2019, 08:36 Tao Lv <ta...@apache.org<mailto: > > > ta...@apache.org>> wrote: > > > > > > @Pedro, seems the issue is still open on the master branch. Do you > > > still > > > think we can have your fix on the 1.5.x branch? > > > > > > Progress since last update: > > > 1. We received several more proposals in the github thread [1]. I > > > humbly > > > ask the reporters to pick the fixes to the v1.5.x. I will keep > > > tracking > > > the > > > progress and the healthy status of the release branch. > > > 2. Thanks to @Lai, the licence issue of julia cat image was fixed on > > > the > > > master branch and I opened a PR to pick it to v1.5.x [2]. > > > 3. The GPU OOM issue was fixed on the master branch by @Lin [3] . But > > > there > > > is a problem with porting the fix to v1.5.x branch [4]. > > > > > > Opens: > > > 1. https://github.com/apache/incubator-mxnet/pull/15803 still can > > > not > > > pass > > > the CI; > > > 2. Call for a update from julia folks about the back porting for [5] > > > and > > > [6] > > > 3. License issue of cub and pybind is still open. @Lai opened a PR > > > [7] > > > to > > > update cub submodule but seems it need more effort than just commit > > > id > > > update. I suspect that we cannot finish this work in 1.5.1 patch > > > release. > > > 4. Still no progress for the sidebar issue on web page [8]. > > > 5. Call for a conclusion about fixing the GPU OOM issue in 1.5.1 > > > > > > Besides, I would like to ask if there is any preference for the > > > release > > > timeline of 1.5.1 patch release? Please share so I can propose the > > > time > > > for > > > code freeze. > > > > > > Thanks, > > > -tao > > > > > > [1] https://github.com/apache/incubator-mxnet/issues/15613. > > > [2] https://github.com/apache/incubator-mxnet/pull/16026 > > > [3] https://github.com/apache/incubator-mxnet/pull/15948 > > > [4] https://github.com/apache/incubator-mxnet/pull/15999 > > > [5] https://github.com/apache/incubator-mxnet/pull/15609 > > > [6] https://github.com/apache/incubator-mxnet/pull/15608 > > > [7] https://github.com/apache/incubator-mxnet/pull/15963 > > > [8] https://github.com/apache/incubator-mxnet/issues/15200 > > > > > > On Wed, Aug 28, 2019 at 5:50 AM Pedro Larroy < > > > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> > > > > > > wrote: > > > > > > Ok. I was just asking if we want this fix in 1.5.1 since it > > > addresses > > > crashes using multiprocessing. The problem with cherry picking is > > > that > > > the > > > patch contains the dynamic load change which shouldn't impact > > > anything > > > else > > > but is not supposed to go in a release branch. > > > > > > On Tue, Aug 27, 2019 at 1:19 PM Lin Yuan <apefor...@gmail.com<mailto: > > > apefor...@gmail.com>> > > > wrote: > > > > > > https://github.com/apache/incubator-mxnet/pull/15762 contains > > > some > > > unrelated changes which is being reverted. Please do not cherry > > > pick > > > it > > > yet. > > > > > > On Mon, Aug 26, 2019 at 4:25 PM Pedro Larroy < > > > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> > > > > > > wrote: > > > > > > There's a fix that I did which seems to still produce crashes > > > in > > > 1.5 > > > for > > > some users, which I got notice today and is fixed in master. > > > > > > Might be useful to put in 1.5.1: > > > https://github.com/apache/incubator-mxnet/pull/15762 ? > > > > > > Pedro. > > > > > > On Tue, Aug 20, 2019 at 7:49 AM Tao Lv <ta...@apache.org<mailto: > > > ta...@apache.org>> > > > wrote: > > > > > > Hi dev, > > > > > > Here is an update for the 1.5.1 patch release. > > > > > > 1. Thanks for the effort from whole community, we have cherry > > > picked > > > a > > > bunch of fixes to v1.5.x branch. So far, the branch looks > > > healthy: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity/ > > > 2. https://github.com/apache/incubator-mxnet/pull/15803 > > > cannot > > > pass > > > the > > > CI; > > > 3. I hope julia folks can take a look at the back porting for > > > https://github.com/apache/incubator-mxnet/pull/15609 and > > > https://github.com/apache/incubator-mxnet/pull/15608 - do we > > > still > > > need > > > them? > > > 4. License issue of cub and pybind is still not fixed. We > > > also > > > has > > > a > > > license issue of a cat image in julia examples. > > > https://github.com/apache/incubator-mxnet/issues/15542 > > > 5. Still no progress for the sidebar issue: > > > https://github.com/apache/incubator-mxnet/issues/15200 > > > 6. There is a GPU OOM issue in 1.5.0 release and already root > > > caused > > > by > > > Lin: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/15703#issuecomment-522780492 > > > . > > > We need decide whether we want to get it fixed in the 1.5.1 > > > patch > > > release. > > > > > > Please find details in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status > > > . > > > > > > Thanks, > > > -tao > > > > > > On Mon, Aug 12, 2019 at 9:57 PM Zhao, Patric < > > > patric.z...@intel.com> > > > wrote: > > > > > > Thanks for the explanation, Marco & Tao. Sounds great! > > > > > > -----Original Message----- > > > From: Tao Lv <ta...@apache.org> > > > Sent: Monday, August 12, 2019 9:54 PM > > > To: dev@mxnet.incubator.apache.org > > > Subject: Re: [Discussion] MXNet 1.5.1 release > > > > > > Regarding the open issue, is there default code > > > owner/maintainer? > > > If > > > so, he/she will be the right people to look into the > > > issue. > > > > > > https://github.com/apache/incubator-mxnet/blob/master/CODEOWNERS > > > > > > > > > I have no idea. But the CODEOWNERS is used to receive > > > change > > > notificaitons, > > > not actually indicates the maintainer of a piece of code. > > > > > > Do we have regularly build, run, functionality and > > > performance > > > testing > > > for > > > this release? > > > > > > > > > As Marco mentioned, build, run and functionality of > > > v1.5.x > > > branch > > > are > > > tracked > > > automatically by the CI for each cherry pick pull request > > > and > > > the > > > nightly tests > > > here: > > > http://jenkins.mxnet-ci.amazon- > > > > > > ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity > > > . > > > I see it's healthy so far. > > > > > > For performance, Shufan will track CPU performance with > > > his > > > test > > > suite > > > and > > > send out the report once the branch is frozen. I'm not > > > sure > > > if > > > there > > > are > > > any > > > other performance tests. > > > > > > On Mon, Aug 12, 2019 at 9:36 PM Marco de Abreu > > > <marco.g.ab...@gmail.com> > > > wrote: > > > > > > Hi Patric, > > > > > > CI should automatically pick up the branch and validate > > > it > > > as > > > usual. > > > > > > Best regards, > > > Marco > > > > > > Zhao, Patric <patric.z...@intel.com> schrieb am Mo., > > > 12. > > > Aug. > > > 2019, > > > 15:22: > > > > > > It's great works, Tao 😊 > > > > > > Regarding the open issue, is there default code > > > owner/maintainer? > > > If > > > so, he/she will be the right people to look into the > > > issue. > > > https://github.com/apache/incubator- > > > mxnet/blob/master/CODEOWNERS > > > > > > Do we have regularly build, run, functionality and > > > performance > > > testing > > > for > > > this release? > > > > > > Thanks, > > > > > > --Patric > > > > > > -----Original Message----- > > > From: Tao Lv <ta...@apache.org> > > > Sent: Monday, August 12, 2019 8:59 PM > > > To: dev@mxnet.incubator.apache.org > > > Subject: Re: [Discussion] MXNet 1.5.1 release > > > > > > Update: > > > > > > We're cherry picking fixes from the master to the > > > v1.5.x > > > branch. > > > Some > > > of > > > them are already merged. Please find details on the > > > cwiki > > > page: > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > > > an+a > > > nd+Status > > > > > > > > > There are still 3 opens: > > > 1. Nightly test failure on CI ( > > > > > > https://github.com/apache/incubator-mxnet/issues/15374 > > > ): > > > The > > > issue > > > is > > > still > > > open. I'm wondering if it has been fixed or not. If > > > not, > > > is > > > there > > > anyone > > > working on it? > > > 2. Broken Sidebar on website API for master and > > > 1.5.0 ( > > > > > > https://github.com/apache/incubator-mxnet/issues/15200 > > > ): > > > I > > > don't > > > see > > > any > > > progress on this issue? Do we still want to include > > > it > > > into > > > 1.5.1 > > > patch > > > release? > > > 3. License issues need to be fixed before 1.6 > > > release ( > > > > > > https://github.com/apache/incubator-mxnet/issues/15542 > > > ): > > > Currently > > > the license issue for code and images is partially > > > fixed > > > on > > > the > > > master > > > branch and > > > will be picked to v1.5.x soon. MKLML license issue > > > is > > > pushed > > > out > > > to 1.6 release. But license issue for cub and > > > pybind > > > is > > > still > > > open. > > > > > > Let me know if you any suggestion. Thanks for your > > > support! > > > > > > -tao > > > > > > > > > On Wed, Aug 7, 2019 at 11:03 PM Tao Lv < > > > ta...@apache.org > > > > > > wrote: > > > > > > > > > Update: > > > > > > Thanks to wkcn's report, Issue #15774 [1] and the > > > fix > > > #15751 > > > [2] > > > are added to the scope of 1.5.1 patch release. > > > For issue #15703 [3], I'm still waiting from the > > > response > > > from > > > the reporter. > > > Issue #15431 [4] was closed as false positive > > > report. > > > I also included several MKL-DNN backend issues > > > reported > > > by > > > mxnet > > > users > > > and downstream projects. They are already fixed > > > on > > > the > > > master > > > branch. > > > > > > Please kindly check the full list of issues need > > > be > > > included > > > in > > > the > > > 1.5.1 patch release: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > > > an+a > > > nd+Status > > > > > > For issues which are already fixed on the master > > > branch, > > > we > > > will > > > start > > > to cherry pick the fix commit to the v1.5.x > > > branch. > > > For > > > issues > > > which are still open, we will start to track the > > > fix > > > process. > > > > > > Thanks for your great support. Let me know if you > > > have > > > any > > > questions or concerns. > > > > > > -tao > > > > > > [1] > > > https://github.com/apache/incubator-mxnet/issues/15774 > > > [2] > > > https://github.com/apache/incubator-mxnet/pull/15751 > > > [3] > > > https://github.com/apache/incubator-mxnet/issues/15703 > > > [4] > > > https://github.com/apache/incubator-mxnet/issues/15431 > > > > > > > > > On Tue, Aug 6, 2019 at 2:04 PM Tao Lv < > > > ta...@apache.org> > > > wrote: > > > > > > > > > Per Sam's proposal [1], Issue #15737 [2] and the > > > fix > > > [3] > > > are > > > added > > > to > > > the scope of 1.5.1 patch release. > > > > > > A friendly reminder: the issue proposing will be > > > closed > > > before > > > 11pm > > > 8/7 CST (8am 8/7 PST). After that, we will start > > > to > > > cherry > > > pick > > > fixes > > > to the v1.5.x branch. > > > > > > > > > [1] > > > https://github.com/apache/incubator- > > > mxnet/issues/15613#issuecomment-5 > > > 18430120 [2] > > > > > > https://github.com/apache/incubator-mxnet/issues/15737 > > > [3] > > > https://github.com/apache/incubator-mxnet/pull/15692 > > > > > > On Thu, Aug 1, 2019 at 4:24 PM Tao Lv < > > > ta...@apache.org > > > > > > wrote: > > > > > > Hi Sandeep/Lai, > > > > > > Thank you for the prompt response! > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/15200 > > > is > > > added > > > to > > > the list to track the sidebar issue. > > > > > > On Thu, Aug 1, 2019 at 7:54 AM sandeep > > > krishnamurthy > > > < > > > sandeep.krishn...@gmail.com> wrote: > > > > > > Thank you Tao and Shufan. > > > Sidebar missing bug in API documentation is > > > inconvenience > > > for > > > the > > > user. > > > It > > > would great if we can fix it with 1.5.1 > > > > > > On Wed, Jul 31, 2019, 10:14 AM Lai Wei < > > > roywei...@gmail.com > > > > > > wrote: > > > > > > Hi Tao, > > > > > > Thank you so much for driving it. Currently > > > nightly > > > test > > > on > > > tutorials are > > > failing and it need to be fixed. [3] I have > > > updated > > > the > > > issue[1] and cwiki.[2] > > > > > > [1] > > > https://github.com/apache/incubator-mxnet/issues/15613 > > > [2] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > > > a > > > n+and+Status > > > [3] > > > https://github.com/apache/incubator-mxnet/issues/15374 > > > > > > Best Regards > > > > > > Lai > > > > > > > > > On Wed, Jul 31, 2019 at 8:04 AM Tao Lv < > > > ta...@apache.org> > > > wrote: > > > > > > Hi community, > > > > > > > > > > > > Thanks for the initiative from Sam > > > (samskalicky@github > > > ), > > > we already > > > have a > > > discussion thread [1] on github about the > > > defects > > > and > > > bugs exposed > > > in the > > > 1.5.0 release. > > > > > > Shufan (juliusshufan@github) and I > > > (TaoLv@github) > > > would > > > like > > > to > > > manage > > > the > > > release of 1.5.1. This will be our first > > > debut > > > on > > > the > > > release > > > process, > > > your > > > comments are always valuable. > > > > > > > > > > > > Per the SemVer 2.0 [2], MXNet 1.5.1 will > > > be > > > a > > > patch > > > release which > > > contains > > > backwards-compatible fixes only. > > > > > > I have created a page on cwiki [3] to > > > track > > > the > > > release > > > process > > > and > > > moved > > > the issues and PRs mentioned in the github > > > discussion > > > thread > > > to > > > the > > > page. > > > > > > > > > > > > Here I would like to ask the community to: > > > > > > (1) Raise any other defect or regression > > > you > > > identified > > > in the > > > 1.5.0 release. Please file a github issue > > > for > > > it > > > and > > > note > > > the issue > > > number in > > > this thread; > > > > > > (2) Please comment with one sentence for > > > why > > > you > > > think > > > the issue is critical and must have in the > > > 1.5.1 > > > release; > > > > > > (3) If the issue is already fixed on > > > master > > > branch > > > or > > > already have > > > a PR > > > WIP, please also note the fix commit id or > > > PR > > > number; > > > > > > (4) If the issue is still open and there > > > is > > > no > > > PR > > > WIP, > > > please > > > indicate > > > whether you'd be willing to help it out; > > > > > > (5) Feel free to comment if any other > > > suggestion > > > for > > > the > > > release. > > > > > > > > > > > > I suggest to keep this thread open for one > > > week > > > to > > > collect enough information and proposals > > > before > > > we > > > decide > > > the timeline for the > > > release. > > > So > > > your timely response will be highly > > > appreciated! > > > > > > > > > > > > PS: Sorry to say that even as a committer, > > > this > > > is > > > the > > > first time > > > for me > > > to > > > manage a release. So it would be great if > > > an > > > experienced > > > committer > > > can > > > help > > > to guide the process. > > > > > > > > > > > > -tao > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/incubator-mxnet/issues/15613 > > > > > > [2] https://semver.org/ > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > > > a > > > n+and+Status > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
