@Aaron, Thank you for looking into these two issues. I have removed the #15609 from the scope of 1.5.1. Please let me know if you have any update about #15608.
@Lai, I'm fine with the decision. License issue about MKL-DNN, cub and pybind is moved to next release. @Sam, I also removed the sidebar issue [3] from the scope of 1.5.1. Besides, I notice one of your cherry picks is stopped by the CI. Please take a look at it. Thanks. *Nice progress since the last update:* 1. Per the discussion, we decided to remove #15609, the license issue about MKL-DNN, cub and pybind, and the sidebar issue [3] from the scope of 1.5.1 patch release; 2. 3 fixes [4] [5] [6] were merged into the v1.5.x branch. *Opens (suggested owners are highlighted):* 1. @Aaron is working on #15608 to see if we can have it in v1.5.x; 2. Two cherry pick PRs [7] [8] cannot pass the CI. I have pinged the authors to take a look at the CI failures. 3. @Kellen proposed 5 fixes [9] for TensorRT but till now only 3 are picked to v1.5.x. Please help to confirm if the other 2 are still needed. 4. Sorry that I missed the proposal for fixing the nightly build [10] in previous update. @Lai, can you help to confirm if it's still valid? 5. @Lin please help to make a conclusion for the GPU OOM issue caused by topk regression [11]. If it cannot be addressed on v1.5.x branch, I will remove it from the scope of this release and mark it as a known issue in the release note. Please find the details in https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status . Thanks, -tao [1] https://github.com/apache/incubator-mxnet/pull/15609 [2] https://github.com/apache/incubator-mxnet/pull/15608 [3] https://github.com/apache/incubator-mxnet/issues/15200 [4] https://github.com/apache/incubator-mxnet/pull/16029 [5] https://github.com/apache/incubator-mxnet/pull/16026 [6] https://github.com/apache/incubator-mxnet/pull/16028 [7] https://github.com/apache/incubator-mxnet/pull/15803 [8] https://github.com/apache/incubator-mxnet/pull/16027 [9] https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-520688668 [10] https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-516937546 [11] https://github.com/apache/incubator-mxnet/issues/15703 On Thu, Aug 29, 2019 at 1:06 AM Skalicky, Sam <sska...@amazon.com.invalid> wrote: > Hi Tao, > > I just talked with Aaron, lets leave the sidebar issue for later. > > I created PRs in the v1.5.x branch to cherry pick the fixes into the 1.5.1 > release: > https://github.com/apache/incubator-mxnet/pull/16027 > https://github.com/apache/incubator-mxnet/pull/16028 > > Thanks for your work on this release! > Sam > > On Aug 28, 2019, at 9:35 AM, Lai Wei <roywei...@gmail.com<mailto: > roywei...@gmail.com>> wrote: > > Hi, > > Regrading the license issue[1], we still have item 3, 4, 5 left. > I think it's better to remove them from 1.5.1 release scope and target for > 1.6.0 as it need more time and requires changes that should not go into > patch release. > > > [1] https://github.com/apache/incubator-mxnet/issues/15542 > > Best Regards > > Lai > > > On Wed, Aug 28, 2019 at 9:20 AM Aaron Markham <aaron.s.mark...@gmail.com > <mailto:aaron.s.mark...@gmail.com>> > wrote: > > 5 no. Install page defaults to master so you don't need to pick it. > 6 probably, but there might be other PRs needed. I'd check out the branch > and attempt the install across platforms to be sure. > > On Wed, Aug 28, 2019, 08:55 Tao Lv <ta...@apache.org<mailto: > ta...@apache.org>> wrote: > > Hi Aaron, > > They were proposed to be ported to v1.5.x at the beginning of the > discussion but I didn't see any action for that. So I'm wondering if > they're still needed. I asked for that in the last update on 8/20 but > didn't get a response. > > If they're still needed, I hope someone who is more familiar with Julia > frontend can help to cherry pick the commits to the v1.5.x branch. > > thanks, > -tao > > On Wed, Aug 28, 2019 at 11:43 PM Aaron Markham < > aaron.s.mark...@gmail.com<mailto:aaron.s.mark...@gmail.com>> > wrote: > > I don't see any request for action on the Julia PRs: 5 or 6. > We didn't put the change in right away because we wanted it to not > break > anything. But the changes are needed to make Julia setup more seamless. > > What "update" is needed? > > > On Wed, Aug 28, 2019, 08:36 Tao Lv <ta...@apache.org<mailto: > ta...@apache.org>> wrote: > > @Pedro, seems the issue is still open on the master branch. Do you > still > think we can have your fix on the 1.5.x branch? > > Progress since last update: > 1. We received several more proposals in the github thread [1]. I > humbly > ask the reporters to pick the fixes to the v1.5.x. I will keep > tracking > the > progress and the healthy status of the release branch. > 2. Thanks to @Lai, the licence issue of julia cat image was fixed on > the > master branch and I opened a PR to pick it to v1.5.x [2]. > 3. The GPU OOM issue was fixed on the master branch by @Lin [3] . But > there > is a problem with porting the fix to v1.5.x branch [4]. > > Opens: > 1. https://github.com/apache/incubator-mxnet/pull/15803 still can > not > pass > the CI; > 2. Call for a update from julia folks about the back porting for [5] > and > [6] > 3. License issue of cub and pybind is still open. @Lai opened a PR > [7] > to > update cub submodule but seems it need more effort than just commit > id > update. I suspect that we cannot finish this work in 1.5.1 patch > release. > 4. Still no progress for the sidebar issue on web page [8]. > 5. Call for a conclusion about fixing the GPU OOM issue in 1.5.1 > > Besides, I would like to ask if there is any preference for the > release > timeline of 1.5.1 patch release? Please share so I can propose the > time > for > code freeze. > > Thanks, > -tao > > [1] https://github.com/apache/incubator-mxnet/issues/15613. > [2] https://github.com/apache/incubator-mxnet/pull/16026 > [3] https://github.com/apache/incubator-mxnet/pull/15948 > [4] https://github.com/apache/incubator-mxnet/pull/15999 > [5] https://github.com/apache/incubator-mxnet/pull/15609 > [6] https://github.com/apache/incubator-mxnet/pull/15608 > [7] https://github.com/apache/incubator-mxnet/pull/15963 > [8] https://github.com/apache/incubator-mxnet/issues/15200 > > On Wed, Aug 28, 2019 at 5:50 AM Pedro Larroy < > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> > > wrote: > > Ok. I was just asking if we want this fix in 1.5.1 since it > addresses > crashes using multiprocessing. The problem with cherry picking is > that > the > patch contains the dynamic load change which shouldn't impact > anything > else > but is not supposed to go in a release branch. > > On Tue, Aug 27, 2019 at 1:19 PM Lin Yuan <apefor...@gmail.com<mailto: > apefor...@gmail.com>> > wrote: > > https://github.com/apache/incubator-mxnet/pull/15762 contains > some > unrelated changes which is being reverted. Please do not cherry > pick > it > yet. > > On Mon, Aug 26, 2019 at 4:25 PM Pedro Larroy < > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> > > wrote: > > There's a fix that I did which seems to still produce crashes > in > 1.5 > for > some users, which I got notice today and is fixed in master. > > Might be useful to put in 1.5.1: > https://github.com/apache/incubator-mxnet/pull/15762 ? > > Pedro. > > On Tue, Aug 20, 2019 at 7:49 AM Tao Lv <ta...@apache.org<mailto: > ta...@apache.org>> > wrote: > > Hi dev, > > Here is an update for the 1.5.1 patch release. > > 1. Thanks for the effort from whole community, we have cherry > picked > a > bunch of fixes to v1.5.x branch. So far, the branch looks > healthy: > > > > > > > > > > http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity/ > 2. https://github.com/apache/incubator-mxnet/pull/15803 > cannot > pass > the > CI; > 3. I hope julia folks can take a look at the back porting for > https://github.com/apache/incubator-mxnet/pull/15609 and > https://github.com/apache/incubator-mxnet/pull/15608 - do we > still > need > them? > 4. License issue of cub and pybind is still not fixed. We > also > has > a > license issue of a cat image in julia examples. > https://github.com/apache/incubator-mxnet/issues/15542 > 5. Still no progress for the sidebar issue: > https://github.com/apache/incubator-mxnet/issues/15200 > 6. There is a GPU OOM issue in 1.5.0 release and already root > caused > by > Lin: > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/15703#issuecomment-522780492 > . > We need decide whether we want to get it fixed in the 1.5.1 > patch > release. > > Please find details in > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status > . > > Thanks, > -tao > > On Mon, Aug 12, 2019 at 9:57 PM Zhao, Patric < > patric.z...@intel.com> > wrote: > > Thanks for the explanation, Marco & Tao. Sounds great! > > -----Original Message----- > From: Tao Lv <ta...@apache.org> > Sent: Monday, August 12, 2019 9:54 PM > To: dev@mxnet.incubator.apache.org > Subject: Re: [Discussion] MXNet 1.5.1 release > > Regarding the open issue, is there default code > owner/maintainer? > If > so, he/she will be the right people to look into the > issue. > > https://github.com/apache/incubator-mxnet/blob/master/CODEOWNERS > > > I have no idea. But the CODEOWNERS is used to receive > change > notificaitons, > not actually indicates the maintainer of a piece of code. > > Do we have regularly build, run, functionality and > performance > testing > for > this release? > > > As Marco mentioned, build, run and functionality of > v1.5.x > branch > are > tracked > automatically by the CI for each cherry pick pull request > and > the > nightly tests > here: > http://jenkins.mxnet-ci.amazon- > > ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity > . > I see it's healthy so far. > > For performance, Shufan will track CPU performance with > his > test > suite > and > send out the report once the branch is frozen. I'm not > sure > if > there > are > any > other performance tests. > > On Mon, Aug 12, 2019 at 9:36 PM Marco de Abreu > <marco.g.ab...@gmail.com> > wrote: > > Hi Patric, > > CI should automatically pick up the branch and validate > it > as > usual. > > Best regards, > Marco > > Zhao, Patric <patric.z...@intel.com> schrieb am Mo., > 12. > Aug. > 2019, > 15:22: > > It's great works, Tao 😊 > > Regarding the open issue, is there default code > owner/maintainer? > If > so, he/she will be the right people to look into the > issue. > https://github.com/apache/incubator- > mxnet/blob/master/CODEOWNERS > > Do we have regularly build, run, functionality and > performance > testing > for > this release? > > Thanks, > > --Patric > > -----Original Message----- > From: Tao Lv <ta...@apache.org> > Sent: Monday, August 12, 2019 8:59 PM > To: dev@mxnet.incubator.apache.org > Subject: Re: [Discussion] MXNet 1.5.1 release > > Update: > > We're cherry picking fixes from the master to the > v1.5.x > branch. > Some > of > them are already merged. Please find details on the > cwiki > page: > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > an+a > nd+Status > > > There are still 3 opens: > 1. Nightly test failure on CI ( > > https://github.com/apache/incubator-mxnet/issues/15374 > ): > The > issue > is > still > open. I'm wondering if it has been fixed or not. If > not, > is > there > anyone > working on it? > 2. Broken Sidebar on website API for master and > 1.5.0 ( > > https://github.com/apache/incubator-mxnet/issues/15200 > ): > I > don't > see > any > progress on this issue? Do we still want to include > it > into > 1.5.1 > patch > release? > 3. License issues need to be fixed before 1.6 > release ( > > https://github.com/apache/incubator-mxnet/issues/15542 > ): > Currently > the license issue for code and images is partially > fixed > on > the > master > branch and > will be picked to v1.5.x soon. MKLML license issue > is > pushed > out > to 1.6 release. But license issue for cub and > pybind > is > still > open. > > Let me know if you any suggestion. Thanks for your > support! > > -tao > > > On Wed, Aug 7, 2019 at 11:03 PM Tao Lv < > ta...@apache.org > > wrote: > > > Update: > > Thanks to wkcn's report, Issue #15774 [1] and the > fix > #15751 > [2] > are added to the scope of 1.5.1 patch release. > For issue #15703 [3], I'm still waiting from the > response > from > the reporter. > Issue #15431 [4] was closed as false positive > report. > I also included several MKL-DNN backend issues > reported > by > mxnet > users > and downstream projects. They are already fixed > on > the > master > branch. > > Please kindly check the full list of issues need > be > included > in > the > 1.5.1 patch release: > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > an+a > nd+Status > > For issues which are already fixed on the master > branch, > we > will > start > to cherry pick the fix commit to the v1.5.x > branch. > For > issues > which are still open, we will start to track the > fix > process. > > Thanks for your great support. Let me know if you > have > any > questions or concerns. > > -tao > > [1] > https://github.com/apache/incubator-mxnet/issues/15774 > [2] > https://github.com/apache/incubator-mxnet/pull/15751 > [3] > https://github.com/apache/incubator-mxnet/issues/15703 > [4] > https://github.com/apache/incubator-mxnet/issues/15431 > > > On Tue, Aug 6, 2019 at 2:04 PM Tao Lv < > ta...@apache.org> > wrote: > > > Per Sam's proposal [1], Issue #15737 [2] and the > fix > [3] > are > added > to > the scope of 1.5.1 patch release. > > A friendly reminder: the issue proposing will be > closed > before > 11pm > 8/7 CST (8am 8/7 PST). After that, we will start > to > cherry > pick > fixes > to the v1.5.x branch. > > > [1] > https://github.com/apache/incubator- > mxnet/issues/15613#issuecomment-5 > 18430120 [2] > > https://github.com/apache/incubator-mxnet/issues/15737 > [3] > https://github.com/apache/incubator-mxnet/pull/15692 > > On Thu, Aug 1, 2019 at 4:24 PM Tao Lv < > ta...@apache.org > > wrote: > > Hi Sandeep/Lai, > > Thank you for the prompt response! > > > https://github.com/apache/incubator-mxnet/issues/15200 > is > added > to > the list to track the sidebar issue. > > On Thu, Aug 1, 2019 at 7:54 AM sandeep > krishnamurthy > < > sandeep.krishn...@gmail.com> wrote: > > Thank you Tao and Shufan. > Sidebar missing bug in API documentation is > inconvenience > for > the > user. > It > would great if we can fix it with 1.5.1 > > On Wed, Jul 31, 2019, 10:14 AM Lai Wei < > roywei...@gmail.com > > wrote: > > Hi Tao, > > Thank you so much for driving it. Currently > nightly > test > on > tutorials are > failing and it need to be fixed. [3] I have > updated > the > issue[1] and cwiki.[2] > > [1] > https://github.com/apache/incubator-mxnet/issues/15613 > [2] > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > a > n+and+Status > [3] > https://github.com/apache/incubator-mxnet/issues/15374 > > Best Regards > > Lai > > > On Wed, Jul 31, 2019 at 8:04 AM Tao Lv < > ta...@apache.org> > wrote: > > Hi community, > > > > Thanks for the initiative from Sam > (samskalicky@github > ), > we already > have a > discussion thread [1] on github about the > defects > and > bugs exposed > in the > 1.5.0 release. > > Shufan (juliusshufan@github) and I > (TaoLv@github) > would > like > to > manage > the > release of 1.5.1. This will be our first > debut > on > the > release > process, > your > comments are always valuable. > > > > Per the SemVer 2.0 [2], MXNet 1.5.1 will > be > a > patch > release which > contains > backwards-compatible fixes only. > > I have created a page on cwiki [3] to > track > the > release > process > and > moved > the issues and PRs mentioned in the github > discussion > thread > to > the > page. > > > > Here I would like to ask the community to: > > (1) Raise any other defect or regression > you > identified > in the > 1.5.0 release. Please file a github issue > for > it > and > note > the issue > number in > this thread; > > (2) Please comment with one sentence for > why > you > think > the issue is critical and must have in the > 1.5.1 > release; > > (3) If the issue is already fixed on > master > branch > or > already have > a PR > WIP, please also note the fix commit id or > PR > number; > > (4) If the issue is still open and there > is > no > PR > WIP, > please > indicate > whether you'd be willing to help it out; > > (5) Feel free to comment if any other > suggestion > for > the > release. > > > > I suggest to keep this thread open for one > week > to > collect enough information and proposals > before > we > decide > the timeline for the > release. > So > your timely response will be highly > appreciated! > > > > PS: Sorry to say that even as a committer, > this > is > the > first time > for me > to > manage a release. So it would be great if > an > experienced > committer > can > help > to guide the process. > > > > -tao > > > > [1] > > https://github.com/apache/incubator-mxnet/issues/15613 > > [2] https://semver.org/ > > [3] > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl > a > n+and+Status > > > > > > > > > > > > > > > > >
