Re: Stopping nightly releases to Pypi

[Skalicky, Sam]

Thanks to Chai for actually making the change, the sort order has now been 
reversed as Tao suggested.

Enjoy!
Sam

> On Jan 20, 2020, at 9:15 PM, Tao Lv  wrote:
> 
> Hi Sam,
> 
> Can you simply change the check on L21 in SortTable from `if (x > y)` to
> `if (x < y)`? Seems then we can have the newest build at the top of table.
> 
> Thanks,
> -tao
> 
> On Tue, Jan 14, 2020 at 7:07 AM Lin Yuan  wrote:
> 
>> Awesome work! It's really convenient to have this page.
>> 
>> Two cents:
>> (1) create a link on mxnet page to this one
>> (2) reorder the nightly as Tao suggested. Newest first.
>> 
>> On Mon, Jan 13, 2020 at 10:25 AM Skalicky, Sam >> 
>> wrote:
>> 
>>> Hi All,
>>> 
>>> The html page source is available at the link (view source, its all in a
>>> single html file), if someone wants to make modifications I’ll be happy
>> to
>>> help integrate those changes and get the latest version published in the
>> S3
>>> bucket. Whenever the final location of the nightly builds is identified
>> we
>>> can move/modify the script appropriately.
>>> 
>>> Sam
>>> 
 On Jan 12, 2020, at 5:41 PM, Tao Lv  wrote:
 
 Thank you for the effort, Sam. One minor suggestion: can we sort and
>> put
 the latest build at the top of the table?
 
 -tao
 
 On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <
>> marco.g.ab...@gmail.com>
 wrote:
 
> Hi Sam,
> 
> that's a great idea, thanks! Can you please adjust the script so it
>> uses
> the artifacts that will be published once Shengs PR gets merged?
> 
> Best regards,
> Marco
> 
> Skalicky, Sam  schrieb am So., 12. Jan.
>>> 2020,
> 23:23:
> 
>> Hi dev,
>> 
>> I made an html page that generates the links to the nightly builds
>> available in the public S3 bucket so you don’t have to log into AWS
>> to
> see
>> them.
>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
>> 
>> Keep in mind we only have builds from January 2020 and December 2019
>> so
>> far.
>> 
>> Sam
>> 
>> On Jan 10, 2020, at 3:05 AM, Sheng Zha > zhash...@apache.org>> wrote:
>> 
>> Size of a change doesn't necessarily reflect the time one spends on
>> the
>> navigating the code base and finding the solution. Also, I tend to
> believe
>> that everyone genuinely wants what's best for the project, just from
>> different perspectives.
>> 
>> Let's focus on improving the CD solution so that security concerns
>> can
>>> be
>> addressed too.
>> 
>> -sz
>> 
>> On 2020/01/09 21:57:30, Chris Olivier > > cjolivie...@apache.org>> wrote:
>> If this tiny fix is representative of the bulk of the reasoning
>> behind
> all
>> the the CD churn recently, then this seems to be of some concern.
>> 
>> -Chris
>> 
>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <
>> marco.g.ab...@gmail.com
>> >
>> wrote:
>> 
>> Great, thanks a lot sheng!
>> 
>> -Marco
>> 
>> Sheng Zha mailto:zhash...@apache.org>> schrieb
>> am
>> Do., 9. Jan. 2020, 14:28:
>> 
>> I'm fixing the CD pipeline in
>> https://github.com/apache/incubator-mxnet/pull/17259/files and will
>> update the s3 publish path so that it's friendly for automatically
>> generating such page.
>> 
>> -sz
>> 
>> On 2020/01/06 18:19:52, "Lausen, Leonard" > >
>> wrote:
>> Consider a user finds a bug in a nightly version but we can't narrow
>> down the
>> version of mxnet used as the name is constant over time. Or users
>> wan't
>> to
>> revert back to the previous nightly version installed but don't know
>> which date
>> it was from due to constant name.
>> 
>> Instead I suggest we introduce an autogenerated page like
>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>> 
>> Then "pip install -f URLTOPAGE mxnet" will install the latest
>> available
>> version.
>> Maybe the team maintaining the S3 bucket can reconsider creating such
>> page for
>> the intermediate time until the CD based nighlty build is operating.
>> 
>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>> +1 for a nightly pip with fixed name.
>> 
>> We need this to track mxnet integration with other packages such as
>> Horovod.
>> 
>> Sam, when do you think we can have this nightly build with a fixed
>> name?
>> 
>> Thanks,
>> 
>> Lin
>> 
>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
>> mailto:sska...@amazon.com.invalid>>
>> wrote:
>> 
>> Hi Tao,
>> 
>> We dont have this yet, but we did think about putting the latest
>> wheels in
>> a specific place in the s3 bucket so they are always updated.
>> Initially we
>> decided not to do this since the main MXNet CD should 

Re: MXNet 1.6 as last release with Python 2 support?

[Skalicky, Sam]

Also, it has been reported that pip wheel installation with latest pip version 
20.0.1 breaks installation of MXNet pip wheels which have py2.py3 in the wheel 
name. This breaks all existing released versions. Work around is to install the 
older version of pip "pip install pip==19.3.1”.

Sam

> On Jan 21, 2020, at 4:35 PM, Chung, Alex  wrote:
> 
> +1
> 
> Sincerely,
> 
> Alex Chung
> Senior Product Manager | AWS AI
> 
> 
> From: shiwen hu 
> Sent: Tuesday, January 21, 2020 4:26 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: MXNet 1.6 as last release with Python 2 support?
> 
> +1
> 
> Lai Wei  于2020年1月18日周六 上午2:51写道：
> 
>> +1
>> 
>> 
>> Best Regards
>> 
>> Lai
>> 
>> 
>> On Fri, Jan 17, 2020 at 10:39 AM Lin Yuan  wrote:
>> 
>>> +1
>>> 
>>> On Fri, Jan 17, 2020 at 10:04 AM Xingjian SHI 
>>> wrote:
>>> 
 +1. We should move to support Python>=3.5 only.
 
 Get Outlook for iOS
 
 From: Lausen, Leonard 
 Sent: Friday, January 17, 2020 10:02:30 AM
 To: d...@mxnet.apache.org 
 Subject: Re: MXNet 1.6 as last release with Python 2 support?
 
 If the lazy consensus passes, I believe the minimum Python version
 supported
 would be Python 3.5.
 
 Python 3.5 because it seems to be the minimum Python 3 version tested
>> by
 our CI,
 specifically in the jobs running on Ubuntu 16.04.
 
 Best regards
 Leonard
 
 On Fri, 2020-01-17 at 17:36 +, Lausen, Leonard wrote:
> Dear MXNet community,
> 
> as effective January 1, 2020, no new bug reports, fixes, or changes
>>> will
 be
> made
> to Python 2, and as MXNet 1.6 will be released after January 1,
>> 2020, I
> suggest
> to announce in the MXNet 1.6 release notes that MXNet 1.6 is the last
 release
> supporting Python 2.
> 
> We have previously reached consensus on announcing that Python 2 is
 dropped in
> the next major release (ie. MXNet 2), however, given the delay in 1.6
 release,
> the plan to release 1.7 in the future and that Python 2 is dead
>>> already I
> think
> we can revisit this assumption.
> 
> Advantages are
> - Time savings for developers, as Python 3 standard library contains
>>> more
>  features than Python 2, and it is more efficient to target only 1
 language
>  (Python 3) instead of 2 languages (Python 2 & 3)
> - Simplification and cost savings for CI
> 
> I thus suggest 72h lazy consensus for announcing dropping of Python 2
>>> as
> described above. If you disagree, please veto (send "-1") and we can
 continue
> supporting Python 2 in all 1.x releases as per previous consensus.
>> Note
 that
> at
> the time of previous consensus, no 1.7 release was planned.
> 
> Best regards
> Leonard
 
>>> 
>> 

Re: MXNet 1.6 as last release with Python 2 support?

[Chung, Alex]

+1

Sincerely,

Alex Chung
Senior Product Manager | AWS AI


From: shiwen hu 
Sent: Tuesday, January 21, 2020 4:26 PM
To: dev@mxnet.incubator.apache.org
Subject: Re: MXNet 1.6 as last release with Python 2 support?

+1

Lai Wei  于2020年1月18日周六 上午2:51写道：

> +1
>
>
> Best Regards
>
> Lai
>
>
> On Fri, Jan 17, 2020 at 10:39 AM Lin Yuan  wrote:
>
> > +1
> >
> > On Fri, Jan 17, 2020 at 10:04 AM Xingjian SHI 
> > wrote:
> >
> > > +1. We should move to support Python>=3.5 only.
> > >
> > > Get Outlook for iOS
> > > 
> > > From: Lausen, Leonard 
> > > Sent: Friday, January 17, 2020 10:02:30 AM
> > > To: d...@mxnet.apache.org 
> > > Subject: Re: MXNet 1.6 as last release with Python 2 support?
> > >
> > > If the lazy consensus passes, I believe the minimum Python version
> > > supported
> > > would be Python 3.5.
> > >
> > > Python 3.5 because it seems to be the minimum Python 3 version tested
> by
> > > our CI,
> > > specifically in the jobs running on Ubuntu 16.04.
> > >
> > > Best regards
> > > Leonard
> > >
> > > On Fri, 2020-01-17 at 17:36 +, Lausen, Leonard wrote:
> > > > Dear MXNet community,
> > > >
> > > > as effective January 1, 2020, no new bug reports, fixes, or changes
> > will
> > > be
> > > > made
> > > > to Python 2, and as MXNet 1.6 will be released after January 1,
> 2020, I
> > > > suggest
> > > > to announce in the MXNet 1.6 release notes that MXNet 1.6 is the last
> > > release
> > > > supporting Python 2.
> > > >
> > > > We have previously reached consensus on announcing that Python 2 is
> > > dropped in
> > > > the next major release (ie. MXNet 2), however, given the delay in 1.6
> > > release,
> > > > the plan to release 1.7 in the future and that Python 2 is dead
> > already I
> > > > think
> > > > we can revisit this assumption.
> > > >
> > > > Advantages are
> > > > - Time savings for developers, as Python 3 standard library contains
> > more
> > > >   features than Python 2, and it is more efficient to target only 1
> > > language
> > > >   (Python 3) instead of 2 languages (Python 2 & 3)
> > > > - Simplification and cost savings for CI
> > > >
> > > > I thus suggest 72h lazy consensus for announcing dropping of Python 2
> > as
> > > > described above. If you disagree, please veto (send "-1") and we can
> > > continue
> > > > supporting Python 2 in all 1.x releases as per previous consensus.
> Note
> > > that
> > > > at
> > > > the time of previous consensus, no 1.7 release was planned.
> > > >
> > > > Best regards
> > > > Leonard
> > >
> >
>

Re: MXNet 1.6 as last release with Python 2 support?

[shiwen hu]

+1

Lai Wei  于2020年1月18日周六 上午2:51写道：

> +1
>
>
> Best Regards
>
> Lai
>
>
> On Fri, Jan 17, 2020 at 10:39 AM Lin Yuan  wrote:
>
> > +1
> >
> > On Fri, Jan 17, 2020 at 10:04 AM Xingjian SHI 
> > wrote:
> >
> > > +1. We should move to support Python>=3.5 only.
> > >
> > > Get Outlook for iOS
> > > 
> > > From: Lausen, Leonard 
> > > Sent: Friday, January 17, 2020 10:02:30 AM
> > > To: d...@mxnet.apache.org 
> > > Subject: Re: MXNet 1.6 as last release with Python 2 support?
> > >
> > > If the lazy consensus passes, I believe the minimum Python version
> > > supported
> > > would be Python 3.5.
> > >
> > > Python 3.5 because it seems to be the minimum Python 3 version tested
> by
> > > our CI,
> > > specifically in the jobs running on Ubuntu 16.04.
> > >
> > > Best regards
> > > Leonard
> > >
> > > On Fri, 2020-01-17 at 17:36 +, Lausen, Leonard wrote:
> > > > Dear MXNet community,
> > > >
> > > > as effective January 1, 2020, no new bug reports, fixes, or changes
> > will
> > > be
> > > > made
> > > > to Python 2, and as MXNet 1.6 will be released after January 1,
> 2020, I
> > > > suggest
> > > > to announce in the MXNet 1.6 release notes that MXNet 1.6 is the last
> > > release
> > > > supporting Python 2.
> > > >
> > > > We have previously reached consensus on announcing that Python 2 is
> > > dropped in
> > > > the next major release (ie. MXNet 2), however, given the delay in 1.6
> > > release,
> > > > the plan to release 1.7 in the future and that Python 2 is dead
> > already I
> > > > think
> > > > we can revisit this assumption.
> > > >
> > > > Advantages are
> > > > - Time savings for developers, as Python 3 standard library contains
> > more
> > > >   features than Python 2, and it is more efficient to target only 1
> > > language
> > > >   (Python 3) instead of 2 languages (Python 2 & 3)
> > > > - Simplification and cost savings for CI
> > > >
> > > > I thus suggest 72h lazy consensus for announcing dropping of Python 2
> > as
> > > > described above. If you disagree, please veto (send "-1") and we can
> > > continue
> > > > supporting Python 2 in all 1.x releases as per previous consensus.
> Note
> > > that
> > > > at
> > > > the time of previous consensus, no 1.7 release was planned.
> > > >
> > > > Best regards
> > > > Leonard
> > >
> >
>

[NOTIFICATION] CI Restart

[Lin Yuan]

Dear Community,

Since Jan 14, 2020, our developers have identified frequently occurrence of
test time-out in our CI system. Nick and Pedro have helped to investigate
this random test timeout, however, due to the design of CI system the
failed instances are already reclaimed and not enough logging is captured
to identify the rootcause.

To resume the development cadence promptly and remove the burden of
repeated PR submission from developers, we decided to take the following
steps:

1) Restart the CI jenkins master
2) Modify the CI configuration so that the snapshot will be taken before
the randomly failed CI instance is reclamied.
3) Keep investigating the rootcause of the random test timeout issue.

In the meanwhile, if you already have PRs currently running in the CI
pipeline, please resubmit your PRs to make sure they will run the pipeline
after restart.

We are sorry for any inconvenience caused.

Best Regards,

Lin

Re: Stop redistributing source code of 3rdparty dependencies to avoid licensing issues

[Markus Weimer]

On Mon, Jan 20, 2020 at 1:17 PM Lausen, Leonard
 wrote:
> If I don't misremember, our mentor Markus Weimer suggested at KDD 2019 in a
> conversation that it's easiest to stop bundling non-ASF 3rdparty code.

You do not misremember :-) If possible, it is easiest for everyone
involved to capture the dependencies in a clean way for the build
system to download. That being said, I came to this conclusion in the
relative sane space of Java / Maven projects. I believe that this
isn't the state of the art in C++, and might be difficult to do. Has
anyone looked into whether the dependencies are or could be made
available e.g. as VCpkgs? [0]

The other aspect to consider are situations where the code mxnet
depends on is actually also managed by people in the mxnet community.
I believe a lot of that was the case when mxnet started in the
incubator from dmlc. For those cases, one could consider inverting the
dependency relationship: The ASF / ASL is designed to be the universal
donor of software. Hence, it is unlikely that depending on any ASF
code would be a problem for current dependencies of the DMLC code.

Markus

[0]: https://github.com/Microsoft/vcpkg