fyi: deep learning framework mentions on arXiv over the past 3 months

[Sebastian]

https://twitter.com/fchollet/status/971863128341323776

call for contributions to next MXNet release

[Steffen Rochel]

Hi - I would like to propose the next MXNet release.
Initial draft content is listed at MXNet wiki
.
I would like to call to all contributors to add features you are working on
and would like to see included in the release. Suggested code freeze is
March 30th  with target release by mid April.

Anirudh Subramanian and Chris Olivier have volunteered to co-manage the
release.

Mark your date: we are planing public meetup on Apr 24th in Seattle.
Details to follow.

Regards,
Steffen

Re: Meetup in Seattle

[Steffen Rochel]

Based on doodle   looks like
April 24th is the preferred date for our public meetup in Seattle.
Mark the date and send suggestions what content you are interested on.
More details to follow.

Regards,
Steffen

On Thu, Mar 1, 2018 at 4:19 AM Pedro Larroy 
wrote:

> Mu, I meant a public meetup in an informal atmosphere  were we can have
> casual conversations about the project, brainstorm and interact with
> contributors and interested people outside of the Amazon conference and
> formal corporate environment. Sometimes these kind of meetups are hosted in
> the company offices, but external people are able to attend. This is very
> common in sites like meetup.com Usually there is a presentation or a talk
> about a topic and then there's drinks and food.
>
> Steffen, April 24th would be a day with less overlap with AMLC related
> events, in any case I created a doodle so we can figure out what times work
> for most of the people.
>
> I will open a new thread on the list with a doodle so everyone can vote on
> his preferred slot.
>
> Pedro
>
>
>
> On Thu, Mar 1, 2018 at 3:54 AM, Li, Mu  wrote:
>
> > Hi Pedro,
> >
> > Do you mean public or private meetup? Simon is organizing a mxnet
> workshop
> > during that conference, but only for internal amazon users.
> >
> > Best,
> > Mu
> >
> > > On Feb 28, 2018, at 3:02 PM, Pedro Larroy <
> pedro.larroy.li...@gmail.com>
> > wrote:
> > >
> > > Hi folks
> > >
> > > It would be nice to organize an MXNet community meet up in Seattle
> > > around the 26th of April, since some of us will be around due to an ML
> > > conference happening in those dates.
> > >
> > > For example going for some drinks / dinner.
> > >
> > > Would anyone from the list be interested in joining?
> > >
> > > Pedro.
> > >
> >
>

Publishing Scala Package/namespace change

[Naveen Swamy]

Hi Guys,

I am working on MXNet Scala Inference APIs
 along with another
contributor Roshani. A while back I noticed that we haven't been publishing
the scala package to Maven for a while now(last one being v0.11.1a under
the dmlc namespace).
Currently users have to build the package manually and then use it, this
hinders adoption and also is painful to build everything from source.

I also see that we haven't changed the namespace to org.apache and instead
are still ml.dmlc namespace.

I wanted to seek your opinion about changing the MXNet-Scala package
namespace to org.apache for the Scala package and publish to Maven in the
upcoming release. I understand that this probably breaks the Semver
semantics that is agreed upon, However I would like to point out that the
Scala package has never been published to maven as 1.0 under org.apache.

Open to suggestions.

Thanks, Naveen

Cython Document iteration

[Chris Olivier]

I have generated some cython documentation based upon work so far regarding
build system, profiling, development, debugging, etc.

https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvVrVGoYdiU3hNsqc

-Chris

Re: Cython Document iteration

[Xingjian SHI]

Great! Also, what’s the current status of Cython support in MXNet? I think we 
have the option to use Cython, which is added by Tianqi.

Best,
Xingjian

Get Outlook for iOS
_
From: Chris Olivier 
Sent: Friday, March 9, 2018 10:21 AM
Subject: Cython Document iteration
To: 


I have generated some cython documentation based upon work so far regarding
build system, profiling, development, debugging, etc.

https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvVrVGoYdiU3hNsqc

-Chris

Re: Cython Document iteration

[Tianqi Chen]

The cython part was not actively maintained and currently disabled, but the
code structure can likely be reused

Tianqi

On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI  wrote:

> Great! Also, what’s the current status of Cython support in MXNet? I think
> we have the option to use Cython, which is added by Tianqi.
>
> Best,
> Xingjian
>
> Get Outlook for iOS
> _
> From: Chris Olivier 
> Sent: Friday, March 9, 2018 10:21 AM
> Subject: Cython Document iteration
> To: 
>
>
> I have generated some cython documentation based upon work so far regarding
> build system, profiling, development, debugging, etc.
>
> https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> rVGoYdiU3hNsqc
>
> -Chris
>
>
>

Re: Cython Document iteration

[Chris Olivier]

I didn't really use anything from the old cython stuff yet.  Mostly setting
up build and development system and doing performance testing.

On Fri, Mar 9, 2018 at 10:32 AM, Tianqi Chen 
wrote:

> The cython part was not actively maintained and currently disabled, but the
> code structure can likely be reused
>
> Tianqi
>
> On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI 
> wrote:
>
> > Great! Also, what’s the current status of Cython support in MXNet? I
> think
> > we have the option to use Cython, which is added by Tianqi.
> >
> > Best,
> > Xingjian
> >
> > Get Outlook for iOS
> > _
> > From: Chris Olivier 
> > Sent: Friday, March 9, 2018 10:21 AM
> > Subject: Cython Document iteration
> > To: 
> >
> >
> > I have generated some cython documentation based upon work so far
> regarding
> > build system, profiling, development, debugging, etc.
> >
> > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> > rVGoYdiU3hNsqc
> >
> > -Chris
> >
> >
> >
>

Re: Cython Document iteration

[Chris Olivier]

Eric told me the current cython files "didn't work", so I have not reused
any of the previous cython attempt at this time.





On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI  wrote:

> Great! Also, what’s the current status of Cython support in MXNet? I think
> we have the option to use Cython, which is added by Tianqi.
>
> Best,
> Xingjian
>
> Get Outlook for iOS
> _
> From: Chris Olivier 
> Sent: Friday, March 9, 2018 10:21 AM
> Subject: Cython Document iteration
> To: 
>
>
> I have generated some cython documentation based upon work so far regarding
> build system, profiling, development, debugging, etc.
>
> https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> rVGoYdiU3hNsqc
>
> -Chris
>
>
>

Re: fyi: deep learning framework mentions on arXiv over the past 3 months

[Mu Li]

arXiv mentioning is a good metric for academic adoption. We should improve
it. But I failed to reproduce the twitter results. Can someone help it?

On Fri, Mar 9, 2018 at 2:13 AM, Sebastian  wrote:

> https://twitter.com/fchollet/status/971863128341323776
>

Re: UTF-8 Support for TextParser

[Anirudh]

Hi,

Upon deeper understanding of customer requirement we found out that the
customer uses only ASCII data with MXNet, just that they want the files
containing UTF-8 BOM at the start and files with different control
characters for newline to play well. dmlc-core already supports control
characters for newline.
Since, the UTF-8 BOM in files is a common use case for other users of MXNet
too (for example, saving excel as UTF-8 csv) I will add support for
handling the UTF-8 BOM in dmlc-core.
I won't be working on UTF8CSVParser unless there is a customer requirement
that comes up later on.

Anirudh



On Wed, Feb 28, 2018 at 11:50 PM, Anirudh  wrote:

> Hi Tianqi,
>
> What do you think about adding a separate parser for CSV with UTF8 support
> in dmlc-core? We can then just add a flag in MXNet for UTF8 and use the
> UTF8 or the ASCII parser based on this flag. (This idea was suggested by
> Mu).
>
> I think there will be some small changes required to the base class
> "TextParserBase" as the method "BackFindEndLine" will have more logic in it
> to check for other code-points for line-breaks, which can be refactored.
> This approach will likely retain the performance of the existing ASCII CSV
> Parser, while allowing MXNet users to make the decision w.r.t usability
> with UTF-8 CSV parser / performance with ASCII CSV parser.
>
> Thanks,
> Anirudh
>
>
> On Mon, Feb 26, 2018 at 5:18 PM, Anirudh  wrote:
>
>> Hi Marco,
>>
>> I understand that there needs to be a different discussion on strong
>> dependency of mxnet and dmlc-core and how to fix it.
>>
>> Having said that, I think the goals of dmlc-core and mxnet are somewhat
>> aligned. Posting in the MXNet dev list for this case
>> is a good way to gather feedback from both the communities since I
>> consider the MXNet community to be mostly a superset of the dmlc-core
>> community.
>>
>> Anirudh
>>
>> On Mon, Feb 26, 2018 at 5:00 PM, Subramanian, Anirudh 
>> wrote:
>>
>>> Hi Tianqi,
>>>
>>> The UTF-8 support would enable other formats like CSV more usable.
>>> Otherwise, they have to handle normalizing their data in some way before
>>> using mxnet.
>>> I understand that there is a tradeoff here because of the efficiency
>>> gains from the parser but the expectation of having to normalize their UTF-8
>>> files may turn users away.
>>>
>>> Anirudh
>>>
>>> On 2/26/18, 3:54 PM, "workc...@gmail.com on behalf of Tianqi Chen" <
>>> workc...@gmail.com on behalf of tqc...@cs.washington.edu> wrote:
>>>
>>> Since LibSVM format is only going to involve numbers and possibly
>>> ascii
>>> characters, is there any reason adding UTF-8 support? Note that
>>> generalization always comes with cost of efficiency and there is some
>>> effort spent on making parser fast
>>>
>>> Tianqi
>>>
>>> On Mon, Feb 26, 2018 at 3:38 PM, Anirudh 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Currently there is no UTF-8 Support for LibSVM, LibFM or CSV Text
>>> parsers.
>>> > I am currently working on adding UTF-8 support for Text parsers.
>>> Since C++
>>> > doesn't have a great built-in support for UTF-8, I am looking at
>>> > third-party libraries which provide Unicode support. I am
>>> considering ICU
>>> > currently. Any comments, suggestions, past experience, gotchas
>>> about
>>> > unicode third party libraries or adding unicode support in general
>>> is
>>> > highly appreciated.
>>> >
>>> > I have created an issue about the same:
>>> > https://github.com/dmlc/dmlc-core/issues/372
>>> > Please feel free to reply to this email or comment on the github
>>> issue if
>>> > you have any inputs.
>>> >
>>> > Anirudh
>>> >
>>>
>>>
>>>
>>
>

Re: UTF-8 Support for TextParser

[Chris Olivier]

For this, are you going to run the entire text through a converter, or just
prepend the UTF-8 header to the file (0xEF,0xBB,0xBF)?

On Fri, Mar 9, 2018 at 12:43 PM, Anirudh  wrote:

> Hi,
>
> Upon deeper understanding of customer requirement we found out that the
> customer uses only ASCII data with MXNet, just that they want the files
> containing UTF-8 BOM at the start and files with different control
> characters for newline to play well. dmlc-core already supports control
> characters for newline.
> Since, the UTF-8 BOM in files is a common use case for other users of MXNet
> too (for example, saving excel as UTF-8 csv) I will add support for
> handling the UTF-8 BOM in dmlc-core.
> I won't be working on UTF8CSVParser unless there is a customer requirement
> that comes up later on.
>
> Anirudh
>
>
>
> On Wed, Feb 28, 2018 at 11:50 PM, Anirudh  wrote:
>
> > Hi Tianqi,
> >
> > What do you think about adding a separate parser for CSV with UTF8
> support
> > in dmlc-core? We can then just add a flag in MXNet for UTF8 and use the
> > UTF8 or the ASCII parser based on this flag. (This idea was suggested by
> > Mu).
> >
> > I think there will be some small changes required to the base class
> > "TextParserBase" as the method "BackFindEndLine" will have more logic in
> it
> > to check for other code-points for line-breaks, which can be refactored.
> > This approach will likely retain the performance of the existing ASCII
> CSV
> > Parser, while allowing MXNet users to make the decision w.r.t usability
> > with UTF-8 CSV parser / performance with ASCII CSV parser.
> >
> > Thanks,
> > Anirudh
> >
> >
> > On Mon, Feb 26, 2018 at 5:18 PM, Anirudh  wrote:
> >
> >> Hi Marco,
> >>
> >> I understand that there needs to be a different discussion on strong
> >> dependency of mxnet and dmlc-core and how to fix it.
> >>
> >> Having said that, I think the goals of dmlc-core and mxnet are somewhat
> >> aligned. Posting in the MXNet dev list for this case
> >> is a good way to gather feedback from both the communities since I
> >> consider the MXNet community to be mostly a superset of the dmlc-core
> >> community.
> >>
> >> Anirudh
> >>
> >> On Mon, Feb 26, 2018 at 5:00 PM, Subramanian, Anirudh <
> ani...@amazon.com>
> >> wrote:
> >>
> >>> Hi Tianqi,
> >>>
> >>> The UTF-8 support would enable other formats like CSV more usable.
> >>> Otherwise, they have to handle normalizing their data in some way
> before
> >>> using mxnet.
> >>> I understand that there is a tradeoff here because of the efficiency
> >>> gains from the parser but the expectation of having to normalize their
> UTF-8
> >>> files may turn users away.
> >>>
> >>> Anirudh
> >>>
> >>> On 2/26/18, 3:54 PM, "workc...@gmail.com on behalf of Tianqi Chen" <
> >>> workc...@gmail.com on behalf of tqc...@cs.washington.edu> wrote:
> >>>
> >>> Since LibSVM format is only going to involve numbers and possibly
> >>> ascii
> >>> characters, is there any reason adding UTF-8 support? Note that
> >>> generalization always comes with cost of efficiency and there is
> some
> >>> effort spent on making parser fast
> >>>
> >>> Tianqi
> >>>
> >>> On Mon, Feb 26, 2018 at 3:38 PM, Anirudh 
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Currently there is no UTF-8 Support for LibSVM, LibFM or CSV Text
> >>> parsers.
> >>> > I am currently working on adding UTF-8 support for Text parsers.
> >>> Since C++
> >>> > doesn't have a great built-in support for UTF-8, I am looking at
> >>> > third-party libraries which provide Unicode support. I am
> >>> considering ICU
> >>> > currently. Any comments, suggestions, past experience, gotchas
> >>> about
> >>> > unicode third party libraries or adding unicode support in
> general
> >>> is
> >>> > highly appreciated.
> >>> >
> >>> > I have created an issue about the same:
> >>> > https://github.com/dmlc/dmlc-core/issues/372
> >>> > Please feel free to reply to this email or comment on the github
> >>> issue if
> >>> > you have any inputs.
> >>> >
> >>> > Anirudh
> >>> >
> >>>
> >>>
> >>>
> >>
> >
>

Re: Publishing Scala Package/namespace change

[YiZhi Liu]

+1 for changing the namespace asap. for the maven deploy, we can have
it build along with pip deployment.


2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> Hi Guys,
>
> I am working on MXNet Scala Inference APIs
>  along with another
> contributor Roshani. A while back I noticed that we haven't been publishing
> the scala package to Maven for a while now(last one being v0.11.1a under
> the dmlc namespace).
> Currently users have to build the package manually and then use it, this
> hinders adoption and also is painful to build everything from source.
>
> I also see that we haven't changed the namespace to org.apache and instead
> are still ml.dmlc namespace.
>
> I wanted to seek your opinion about changing the MXNet-Scala package
> namespace to org.apache for the Scala package and publish to Maven in the
> upcoming release. I understand that this probably breaks the Semver
> semantics that is agreed upon, However I would like to point out that the
> Scala package has never been published to maven as 1.0 under org.apache.
>
> Open to suggestions.
>
> Thanks, Naveen



-- 
Yizhi Liu
DMLC member
Amazon Web Services
Vancouver, Canada

Python extension module survey

[Chris Olivier]

If we were to distribute a python extension module, which python versions
would we need to support before requiring:

1) a recompile on the end-user host (note this may not be trivial for MSVC)

2) an automatic download of a prebuilt module from an internet repository
(ie S3 or Apache FTP) at runtime


-Chris

Re: Publishing Scala Package/namespace change

[Marco de Abreu]

While it has never been published, there have still been releases under
Apache and - as you mentioned - customers that build off the source. This
would cause compatibility issues.

In general I actively support the idea of enhancing the Scala package, but
I think that we have to solve another problem first. At the moment, all
APIs are bound to the MXNet core versioning and vica versa.

In my opinion, we should first separate the APIs from the MXNet core, start
versioning them separately and then make changes like these. While it would
be possible (although not right) to make an exception here, we still don't
solve the root problem and we are going to run into the same issues with
the next big API update.

Just to mention another example: our team got a request to rewrite the Cpp
package, but we actually would not be able to merge it into MXNet since it
breaks the existing Cpp package API - means we would need a major version
increase.

We really should solve this problem once and for all, giving back a lot of
freedom and reducing overhead in the long term.

-Marco

YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:

> +1 for changing the namespace asap. for the maven deploy, we can have
> it build along with pip deployment.
>
>
> 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > Hi Guys,
> >
> > I am working on MXNet Scala Inference APIs
> >  along with another
> > contributor Roshani. A while back I noticed that we haven't been
> publishing
> > the scala package to Maven for a while now(last one being v0.11.1a under
> > the dmlc namespace).
> > Currently users have to build the package manually and then use it, this
> > hinders adoption and also is painful to build everything from source.
> >
> > I also see that we haven't changed the namespace to org.apache and
> instead
> > are still ml.dmlc namespace.
> >
> > I wanted to seek your opinion about changing the MXNet-Scala package
> > namespace to org.apache for the Scala package and publish to Maven in the
> > upcoming release. I understand that this probably breaks the Semver
> > semantics that is agreed upon, However I would like to point out that the
> > Scala package has never been published to maven as 1.0 under org.apache.
> >
> > Open to suggestions.
> >
> > Thanks, Naveen
>
>
>
> --
> Yizhi Liu
> DMLC member
> Amazon Web Services
> Vancouver, Canada
>

Re: UTF-8 Support for TextParser

[Anirudh]

Won't run entire text through converter, will just ignore the BOM character
during parsing stage.

Anirudh

On Fri, Mar 9, 2018 at 1:12 PM, Chris Olivier  wrote:

> For this, are you going to run the entire text through a converter, or just
> prepend the UTF-8 header to the file (0xEF,0xBB,0xBF)?
>
> On Fri, Mar 9, 2018 at 12:43 PM, Anirudh  wrote:
>
> > Hi,
> >
> > Upon deeper understanding of customer requirement we found out that the
> > customer uses only ASCII data with MXNet, just that they want the files
> > containing UTF-8 BOM at the start and files with different control
> > characters for newline to play well. dmlc-core already supports control
> > characters for newline.
> > Since, the UTF-8 BOM in files is a common use case for other users of
> MXNet
> > too (for example, saving excel as UTF-8 csv) I will add support for
> > handling the UTF-8 BOM in dmlc-core.
> > I won't be working on UTF8CSVParser unless there is a customer
> requirement
> > that comes up later on.
> >
> > Anirudh
> >
> >
> >
> > On Wed, Feb 28, 2018 at 11:50 PM, Anirudh  wrote:
> >
> > > Hi Tianqi,
> > >
> > > What do you think about adding a separate parser for CSV with UTF8
> > support
> > > in dmlc-core? We can then just add a flag in MXNet for UTF8 and use the
> > > UTF8 or the ASCII parser based on this flag. (This idea was suggested
> by
> > > Mu).
> > >
> > > I think there will be some small changes required to the base class
> > > "TextParserBase" as the method "BackFindEndLine" will have more logic
> in
> > it
> > > to check for other code-points for line-breaks, which can be
> refactored.
> > > This approach will likely retain the performance of the existing ASCII
> > CSV
> > > Parser, while allowing MXNet users to make the decision w.r.t usability
> > > with UTF-8 CSV parser / performance with ASCII CSV parser.
> > >
> > > Thanks,
> > > Anirudh
> > >
> > >
> > > On Mon, Feb 26, 2018 at 5:18 PM, Anirudh 
> wrote:
> > >
> > >> Hi Marco,
> > >>
> > >> I understand that there needs to be a different discussion on strong
> > >> dependency of mxnet and dmlc-core and how to fix it.
> > >>
> > >> Having said that, I think the goals of dmlc-core and mxnet are
> somewhat
> > >> aligned. Posting in the MXNet dev list for this case
> > >> is a good way to gather feedback from both the communities since I
> > >> consider the MXNet community to be mostly a superset of the dmlc-core
> > >> community.
> > >>
> > >> Anirudh
> > >>
> > >> On Mon, Feb 26, 2018 at 5:00 PM, Subramanian, Anirudh <
> > ani...@amazon.com>
> > >> wrote:
> > >>
> > >>> Hi Tianqi,
> > >>>
> > >>> The UTF-8 support would enable other formats like CSV more usable.
> > >>> Otherwise, they have to handle normalizing their data in some way
> > before
> > >>> using mxnet.
> > >>> I understand that there is a tradeoff here because of the efficiency
> > >>> gains from the parser but the expectation of having to normalize
> their
> > UTF-8
> > >>> files may turn users away.
> > >>>
> > >>> Anirudh
> > >>>
> > >>> On 2/26/18, 3:54 PM, "workc...@gmail.com on behalf of Tianqi Chen" <
> > >>> workc...@gmail.com on behalf of tqc...@cs.washington.edu> wrote:
> > >>>
> > >>> Since LibSVM format is only going to involve numbers and possibly
> > >>> ascii
> > >>> characters, is there any reason adding UTF-8 support? Note that
> > >>> generalization always comes with cost of efficiency and there is
> > some
> > >>> effort spent on making parser fast
> > >>>
> > >>> Tianqi
> > >>>
> > >>> On Mon, Feb 26, 2018 at 3:38 PM, Anirudh 
> > >>> wrote:
> > >>>
> > >>> > Hi all,
> > >>> >
> > >>> > Currently there is no UTF-8 Support for LibSVM, LibFM or CSV
> Text
> > >>> parsers.
> > >>> > I am currently working on adding UTF-8 support for Text
> parsers.
> > >>> Since C++
> > >>> > doesn't have a great built-in support for UTF-8, I am looking
> at
> > >>> > third-party libraries which provide Unicode support. I am
> > >>> considering ICU
> > >>> > currently. Any comments, suggestions, past experience, gotchas
> > >>> about
> > >>> > unicode third party libraries or adding unicode support in
> > general
> > >>> is
> > >>> > highly appreciated.
> > >>> >
> > >>> > I have created an issue about the same:
> > >>> > https://github.com/dmlc/dmlc-core/issues/372
> > >>> > Please feel free to reply to this email or comment on the
> github
> > >>> issue if
> > >>> > you have any inputs.
> > >>> >
> > >>> > Anirudh
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>
> > >
> >
>

Re: Python extension module survey

[Marco de Abreu]

I'm a bit lost here. Could you explain a bit more what you would like to do?

-Marco

Chris Olivier  schrieb am Fr., 9. März 2018, 22:46:

> If we were to distribute a python extension module, which python versions
> would we need to support before requiring:
>
> 1) a recompile on the end-user host (note this may not be trivial for MSVC)
>
> 2) an automatic download of a prebuilt module from an internet repository
> (ie S3 or Apache FTP) at runtime
>
>
> -Chris
>

Re: Publishing Scala Package/namespace change

[Naveen Swamy]

MXNet Scala APis already generate a MXNet Core Scala package based off the
cpp backend already. I think customers who are building from source would
love to get Maven package given that it takes so much pain.

Are you suggesting we take MXNet-Scala APIs into a separate release cycle,
it is possible and can start with this one but It would not make a lot of
sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This won't very
different from breaking backward compatibility when we release a new
package.

IMO managing separate release cycles for different language bindings could
turn into a lot of work for the community unnecessarily especially since
they are closely

On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu  wrote:

> While it has never been published, there have still been releases under
> Apache and - as you mentioned - customers that build off the source. This
> would cause compatibility issues.
>
> In general I actively support the idea of enhancing the Scala package, but
> I think that we have to solve another problem first. At the moment, all
> APIs are bound to the MXNet core versioning and vica versa.
>
> In my opinion, we should first separate the APIs from the MXNet core, start
> versioning them separately and then make changes like these. While it would
> be possible (although not right) to make an exception here, we still don't
> solve the root problem and we are going to run into the same issues with
> the next big API update.
>
> Just to mention another example: our team got a request to rewrite the Cpp
> package, but we actually would not be able to merge it into MXNet since it
> breaks the existing Cpp package API - means we would need a major version
> increase.
>
> We really should solve this problem once and for all, giving back a lot of
> freedom and reducing overhead in the long term.
>
> -Marco
>
> YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:
>
> > +1 for changing the namespace asap. for the maven deploy, we can have
> > it build along with pip deployment.
> >
> >
> > 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > > Hi Guys,
> > >
> > > I am working on MXNet Scala Inference APIs
> > >  along with another
> > > contributor Roshani. A while back I noticed that we haven't been
> > publishing
> > > the scala package to Maven for a while now(last one being v0.11.1a
> under
> > > the dmlc namespace).
> > > Currently users have to build the package manually and then use it,
> this
> > > hinders adoption and also is painful to build everything from source.
> > >
> > > I also see that we haven't changed the namespace to org.apache and
> > instead
> > > are still ml.dmlc namespace.
> > >
> > > I wanted to seek your opinion about changing the MXNet-Scala package
> > > namespace to org.apache for the Scala package and publish to Maven in
> the
> > > upcoming release. I understand that this probably breaks the Semver
> > > semantics that is agreed upon, However I would like to point out that
> the
> > > Scala package has never been published to maven as 1.0 under
> org.apache.
> > >
> > > Open to suggestions.
> > >
> > > Thanks, Naveen
> >
> >
> >
> > --
> > Yizhi Liu
> > DMLC member
> > Amazon Web Services
> > Vancouver, Canada
> >
>

Re: Publishing Scala Package/namespace change

[Marco de Abreu]

I agree, separate release cycles would cause a lot of hassle and grant now
visible benefit, I'd bind these to the main release cycles. But I would
separate the versioning of each package.

At the moment, we are bound by the least common denominator in terms of API
changes and versions. While this gives stability to our users, it limits us
in our development since we have to consider backwards compatibility for
more than half a dozen languages. Especially considering this being an open
source project, it's hard to manage this without significant constraints.

With that proposal we would have the core of MXNet as some type of hardened
and "compact" but fully functional and backwards  compatible kernel with a
C/C++ API. This would be the main version of MXNet. If there is suddenly a
big community effort to heavily improve a certain API (like Scala), we
would not block them with semver being the reason but just increase the
version of that specific API. This gives us flexibility and our customers
the possibility to benefit from an extensively improved API.

Something we also could consider with this clear separation would be to
give the possibility to have a legacy API for some time and basically offer
two versions. We would be maintaining both for a few months to give
customers the possibility to migrate and then remove it. At the moment, a
layout like this is hard to pull off.

I think we should really work on this topic as a community and provide a
good and maintainable solution. In the end, it's a huge plus (that is being
appreciated by our users) for MXNet that we are offering such a variety of
APIs. Now we need a concept to maintain them properly.

-Marco

Naveen Swamy  schrieb am Fr., 9. März 2018, 23:10:

> MXNet Scala APis already generate a MXNet Core Scala package based off the
> cpp backend already. I think customers who are building from source would
> love to get Maven package given that it takes so much pain.
>
> Are you suggesting we take MXNet-Scala APIs into a separate release cycle,
> it is possible and can start with this one but It would not make a lot of
> sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This won't very
> different from breaking backward compatibility when we release a new
> package.
>
> IMO managing separate release cycles for different language bindings could
> turn into a lot of work for the community unnecessarily especially since
> they are closely
>
> On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu <
> marco.g.ab...@googlemail.com
> > wrote:
>
> > While it has never been published, there have still been releases under
> > Apache and - as you mentioned - customers that build off the source. This
> > would cause compatibility issues.
> >
> > In general I actively support the idea of enhancing the Scala package,
> but
> > I think that we have to solve another problem first. At the moment, all
> > APIs are bound to the MXNet core versioning and vica versa.
> >
> > In my opinion, we should first separate the APIs from the MXNet core,
> start
> > versioning them separately and then make changes like these. While it
> would
> > be possible (although not right) to make an exception here, we still
> don't
> > solve the root problem and we are going to run into the same issues with
> > the next big API update.
> >
> > Just to mention another example: our team got a request to rewrite the
> Cpp
> > package, but we actually would not be able to merge it into MXNet since
> it
> > breaks the existing Cpp package API - means we would need a major version
> > increase.
> >
> > We really should solve this problem once and for all, giving back a lot
> of
> > freedom and reducing overhead in the long term.
> >
> > -Marco
> >
> > YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:
> >
> > > +1 for changing the namespace asap. for the maven deploy, we can have
> > > it build along with pip deployment.
> > >
> > >
> > > 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > > > Hi Guys,
> > > >
> > > > I am working on MXNet Scala Inference APIs
> > > >  along with another
> > > > contributor Roshani. A while back I noticed that we haven't been
> > > publishing
> > > > the scala package to Maven for a while now(last one being v0.11.1a
> > under
> > > > the dmlc namespace).
> > > > Currently users have to build the package manually and then use it,
> > this
> > > > hinders adoption and also is painful to build everything from source.
> > > >
> > > > I also see that we haven't changed the namespace to org.apache and
> > > instead
> > > > are still ml.dmlc namespace.
> > > >
> > > > I wanted to seek your opinion about changing the MXNet-Scala package
> > > > namespace to org.apache for the Scala package and publish to Maven in
> > the
> > > > upcoming release. I understand that this probably breaks the Semver
> > > > semantics that is agreed upon, However I would like to point out that
> > the
> > > > Scala package has never been published to mave

Re: Publishing Scala Package/namespace change

[Chris Olivier]

IMHO, I don't think having separate versions and releases for the scala API
will work for the following reasons:

   - Scala API is not its own Apache project, so we don't really have a
   mechanism to release it separately and not the manpower of volunteers to
   maintain all the BS that goes along with releases
   - We operate under the assumption (which is fair, I think) that the
   Scala API is only compatible with the exact C API for which it was built
   against.  Since the Scala API ships with its own libmxnet.so, and
   libmxnet.so is "stable" only at release boundaries, then it would be risky
   to pick arbitrary points in the Scala evolution as "new versions" and birth
   them into the World with just whatever mxnet commit was at the top at that
   time

I am also in favor of changing the namespace ASAP before the massive
adoption of the Scala API that's about to happen, because it'll be way
harder to do later.

-Chris


On Fri, Mar 9, 2018 at 2:09 PM, Naveen Swamy  wrote:

> MXNet Scala APis already generate a MXNet Core Scala package based off the
> cpp backend already. I think customers who are building from source would
> love to get Maven package given that it takes so much pain.
>
> Are you suggesting we take MXNet-Scala APIs into a separate release cycle,
> it is possible and can start with this one but It would not make a lot of
> sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This won't very
> different from breaking backward compatibility when we release a new
> package.
>
> IMO managing separate release cycles for different language bindings could
> turn into a lot of work for the community unnecessarily especially since
> they are closely
>
> On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu <
> marco.g.ab...@googlemail.com
> > wrote:
>
> > While it has never been published, there have still been releases under
> > Apache and - as you mentioned - customers that build off the source. This
> > would cause compatibility issues.
> >
> > In general I actively support the idea of enhancing the Scala package,
> but
> > I think that we have to solve another problem first. At the moment, all
> > APIs are bound to the MXNet core versioning and vica versa.
> >
> > In my opinion, we should first separate the APIs from the MXNet core,
> start
> > versioning them separately and then make changes like these. While it
> would
> > be possible (although not right) to make an exception here, we still
> don't
> > solve the root problem and we are going to run into the same issues with
> > the next big API update.
> >
> > Just to mention another example: our team got a request to rewrite the
> Cpp
> > package, but we actually would not be able to merge it into MXNet since
> it
> > breaks the existing Cpp package API - means we would need a major version
> > increase.
> >
> > We really should solve this problem once and for all, giving back a lot
> of
> > freedom and reducing overhead in the long term.
> >
> > -Marco
> >
> > YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:
> >
> > > +1 for changing the namespace asap. for the maven deploy, we can have
> > > it build along with pip deployment.
> > >
> > >
> > > 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > > > Hi Guys,
> > > >
> > > > I am working on MXNet Scala Inference APIs
> > > >  along with another
> > > > contributor Roshani. A while back I noticed that we haven't been
> > > publishing
> > > > the scala package to Maven for a while now(last one being v0.11.1a
> > under
> > > > the dmlc namespace).
> > > > Currently users have to build the package manually and then use it,
> > this
> > > > hinders adoption and also is painful to build everything from source.
> > > >
> > > > I also see that we haven't changed the namespace to org.apache and
> > > instead
> > > > are still ml.dmlc namespace.
> > > >
> > > > I wanted to seek your opinion about changing the MXNet-Scala package
> > > > namespace to org.apache for the Scala package and publish to Maven in
> > the
> > > > upcoming release. I understand that this probably breaks the Semver
> > > > semantics that is agreed upon, However I would like to point out that
> > the
> > > > Scala package has never been published to maven as 1.0 under
> > org.apache.
> > > >
> > > > Open to suggestions.
> > > >
> > > > Thanks, Naveen
> > >
> > >
> > >
> > > --
> > > Yizhi Liu
> > > DMLC member
> > > Amazon Web Services
> > > Vancouver, Canada
> > >
> >
>

Re: Publishing Scala Package/namespace change

[Nan Zhu]

I think last time we postpone it because the release is a minor version

but actually such a change is actually affordable for a jump from 1.1 - 1.2

-1 on separate versions (not following apache rules)


On Fri, Mar 9, 2018 at 2:38 PM, Chris Olivier  wrote:

> IMHO, I don't think having separate versions and releases for the scala API
> will work for the following reasons:
>
>- Scala API is not its own Apache project, so we don't really have a
>mechanism to release it separately and not the manpower of volunteers to
>maintain all the BS that goes along with releases
>- We operate under the assumption (which is fair, I think) that the
>Scala API is only compatible with the exact C API for which it was built
>against.  Since the Scala API ships with its own libmxnet.so, and
>libmxnet.so is "stable" only at release boundaries, then it would be
> risky
>to pick arbitrary points in the Scala evolution as "new versions" and
> birth
>them into the World with just whatever mxnet commit was at the top at
> that
>time
>
> I am also in favor of changing the namespace ASAP before the massive
> adoption of the Scala API that's about to happen, because it'll be way
> harder to do later.
>
> -Chris
>
>
> On Fri, Mar 9, 2018 at 2:09 PM, Naveen Swamy  wrote:
>
> > MXNet Scala APis already generate a MXNet Core Scala package based off
> the
> > cpp backend already. I think customers who are building from source would
> > love to get Maven package given that it takes so much pain.
> >
> > Are you suggesting we take MXNet-Scala APIs into a separate release
> cycle,
> > it is possible and can start with this one but It would not make a lot of
> > sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This won't
> very
> > different from breaking backward compatibility when we release a new
> > package.
> >
> > IMO managing separate release cycles for different language bindings
> could
> > turn into a lot of work for the community unnecessarily especially since
> > they are closely
> >
> > On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu <
> > marco.g.ab...@googlemail.com
> > > wrote:
> >
> > > While it has never been published, there have still been releases under
> > > Apache and - as you mentioned - customers that build off the source.
> This
> > > would cause compatibility issues.
> > >
> > > In general I actively support the idea of enhancing the Scala package,
> > but
> > > I think that we have to solve another problem first. At the moment, all
> > > APIs are bound to the MXNet core versioning and vica versa.
> > >
> > > In my opinion, we should first separate the APIs from the MXNet core,
> > start
> > > versioning them separately and then make changes like these. While it
> > would
> > > be possible (although not right) to make an exception here, we still
> > don't
> > > solve the root problem and we are going to run into the same issues
> with
> > > the next big API update.
> > >
> > > Just to mention another example: our team got a request to rewrite the
> > Cpp
> > > package, but we actually would not be able to merge it into MXNet since
> > it
> > > breaks the existing Cpp package API - means we would need a major
> version
> > > increase.
> > >
> > > We really should solve this problem once and for all, giving back a lot
> > of
> > > freedom and reducing overhead in the long term.
> > >
> > > -Marco
> > >
> > > YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:
> > >
> > > > +1 for changing the namespace asap. for the maven deploy, we can have
> > > > it build along with pip deployment.
> > > >
> > > >
> > > > 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > > > > Hi Guys,
> > > > >
> > > > > I am working on MXNet Scala Inference APIs
> > > > >  along with
> another
> > > > > contributor Roshani. A while back I noticed that we haven't been
> > > > publishing
> > > > > the scala package to Maven for a while now(last one being v0.11.1a
> > > under
> > > > > the dmlc namespace).
> > > > > Currently users have to build the package manually and then use it,
> > > this
> > > > > hinders adoption and also is painful to build everything from
> source.
> > > > >
> > > > > I also see that we haven't changed the namespace to org.apache and
> > > > instead
> > > > > are still ml.dmlc namespace.
> > > > >
> > > > > I wanted to seek your opinion about changing the MXNet-Scala
> package
> > > > > namespace to org.apache for the Scala package and publish to Maven
> in
> > > the
> > > > > upcoming release. I understand that this probably breaks the Semver
> > > > > semantics that is agreed upon, However I would like to point out
> that
> > > the
> > > > > Scala package has never been published to maven as 1.0 under
> > > org.apache.
> > > > >
> > > > > Open to suggestions.
> > > > >
> > > > > Thanks, Naveen
> > > >
> > > >
> > > >
> > > > --
> > > > Yizhi Liu
> > > > DMLC member
> > > > Amazon Web Services
> > > > Vancouver, Canada
> > > >
> > >

Re: Refactoring docker handling on CI and locally

[Marco de Abreu]

Hello,

fyi, this change has just been merged to master and PRs containing
modifications to Dockerfiles will have to be adopted to the new layout.

I will write the documentation on Monday, the issue can be tracked at
https://issues.apache.org/jira/browse/MXNET-71. Please don't hesitate to
reach out to me if any issues arise or in case you have questions.

Best regards,
Marco

On Fri, Mar 2, 2018 at 11:27 AM, Pedro Larroy 
wrote:

> This is also part of clearing the path to integrate IoT devices into the
> testing environment.
>
> In addition we have too much logic in the Jenkinsfile which is an
> anti-pattern. Most of what CI does will be reproducible locally with a
> simple command, which I think is very good to test PRs or local changes.
>
> @Tianqi thanks for your comments in the PR.
>
>
> On Thu, Mar 1, 2018 at 11:43 PM, Marco de Abreu <
> marco.g.ab...@googlemail.com> wrote:
>
> > Exactly. The current implementation using ci_build.sh is tightly
> integrated
> > into system, especially due to  /tests/ci_build/with_the_same_user that
> > can
> > only be executed on Ubuntu as container OS. Since we'll add CentOS 7 to
> our
> > supported OS, this change was necessary. We also used that moment to
> > restructure the Dockerfiles to allow better maintainability.
> >
> > In terms of the OS the docker host is using, we will still be targetting
> > Ubuntu 16.04, but we're also going to establish compatibility with Mac.
> > Docker on Windows might work, but I don't it's time investing time into
> > exploring towards that direction.
> >
> > -Marco
> >
> > On Thu, Mar 1, 2018 at 11:29 PM, Tianqi Chen 
> > wrote:
> >
> > > Just to be clear, is this mainly about rewriting ci_build.sh into
> python
> > so
> > > that it is more portable across platforms?
> > >
> > > Tianqi
> > >
> > > On Thu, Mar 1, 2018 at 2:11 PM, Marco de Abreu <
> > > marco.g.ab...@googlemail.com
> > > > wrote:
> > >
> > > > Hello,
> > > >
> > > > we have identified issues with the way CI handles docker containers
> and
> > > > thus started a rewrite of this component.
> > > >
> > > > Besides problems that limit the maintainability of our CI system, the
> > > > current solution is not extensible in terms of other operating
> systems
> > > > running inside the container. But more importantly, developers are
> not
> > > able
> > > > to reproduce CI issues locally on their computer. We would like to
> give
> > > > every developer the chance to reproduce problems locally without
> going
> > > > through an iterative trial & error process assisted by our CI
> system. I
> > > > will follow up with more documentation at a later point in stage.
> > > >
> > > > This project will change the structure of our dockerfiles, remove a
> lot
> > > of
> > > > code from the Jenkinsfile, centralize the configuration and behaviour
> > as
> > > > well as clean up some files. Please expect some merge conflicts after
> > > this
> > > > PR has been shipped. Right now we're in the process of migrating all
> > jobs
> > > > to the new layout; you can track the progress at
> > > > https://github.com/apache/incubator-mxnet/pull/9946.
> > > >
> > > > Please don't hesitate to reach out to me in case of any questions.
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > >
> >
>

Re: Publishing Scala Package/namespace change

[Marco de Abreu]

My proposal was to have separate versioning but same release cycles -
having separate cycles was just an idea.

So how would we approach this problem in future if there's a big
improvement on one of our language bindings? Like I said previously, it
might be fine for this case, but we will run into the same issues for the
next time and it's not a great user experience if we just do this
"silently", provide no transitional solution and just replace an existing
API with a new improved one.

I would like to have a solution for this problem before we move on.

-Marco



On Sat, Mar 10, 2018 at 12:06 AM, Nan Zhu  wrote:

> I think last time we postpone it because the release is a minor version
>
> but actually such a change is actually affordable for a jump from 1.1 - 1.2
>
> -1 on separate versions (not following apache rules)
>
>
> On Fri, Mar 9, 2018 at 2:38 PM, Chris Olivier 
> wrote:
>
> > IMHO, I don't think having separate versions and releases for the scala
> API
> > will work for the following reasons:
> >
> >- Scala API is not its own Apache project, so we don't really have a
> >mechanism to release it separately and not the manpower of volunteers
> to
> >maintain all the BS that goes along with releases
> >- We operate under the assumption (which is fair, I think) that the
> >Scala API is only compatible with the exact C API for which it was
> built
> >against.  Since the Scala API ships with its own libmxnet.so, and
> >libmxnet.so is "stable" only at release boundaries, then it would be
> > risky
> >to pick arbitrary points in the Scala evolution as "new versions" and
> > birth
> >them into the World with just whatever mxnet commit was at the top at
> > that
> >time
> >
> > I am also in favor of changing the namespace ASAP before the massive
> > adoption of the Scala API that's about to happen, because it'll be way
> > harder to do later.
> >
> > -Chris
> >
> >
> > On Fri, Mar 9, 2018 at 2:09 PM, Naveen Swamy  wrote:
> >
> > > MXNet Scala APis already generate a MXNet Core Scala package based off
> > the
> > > cpp backend already. I think customers who are building from source
> would
> > > love to get Maven package given that it takes so much pain.
> > >
> > > Are you suggesting we take MXNet-Scala APIs into a separate release
> > cycle,
> > > it is possible and can start with this one but It would not make a lot
> of
> > > sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This won't
> > very
> > > different from breaking backward compatibility when we release a new
> > > package.
> > >
> > > IMO managing separate release cycles for different language bindings
> > could
> > > turn into a lot of work for the community unnecessarily especially
> since
> > > they are closely
> > >
> > > On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu <
> > > marco.g.ab...@googlemail.com
> > > > wrote:
> > >
> > > > While it has never been published, there have still been releases
> under
> > > > Apache and - as you mentioned - customers that build off the source.
> > This
> > > > would cause compatibility issues.
> > > >
> > > > In general I actively support the idea of enhancing the Scala
> package,
> > > but
> > > > I think that we have to solve another problem first. At the moment,
> all
> > > > APIs are bound to the MXNet core versioning and vica versa.
> > > >
> > > > In my opinion, we should first separate the APIs from the MXNet core,
> > > start
> > > > versioning them separately and then make changes like these. While it
> > > would
> > > > be possible (although not right) to make an exception here, we still
> > > don't
> > > > solve the root problem and we are going to run into the same issues
> > with
> > > > the next big API update.
> > > >
> > > > Just to mention another example: our team got a request to rewrite
> the
> > > Cpp
> > > > package, but we actually would not be able to merge it into MXNet
> since
> > > it
> > > > breaks the existing Cpp package API - means we would need a major
> > version
> > > > increase.
> > > >
> > > > We really should solve this problem once and for all, giving back a
> lot
> > > of
> > > > freedom and reducing overhead in the long term.
> > > >
> > > > -Marco
> > > >
> > > > YiZhi Liu  schrieb am Fr., 9. März 2018, 22:44:
> > > >
> > > > > +1 for changing the namespace asap. for the maven deploy, we can
> have
> > > > > it build along with pip deployment.
> > > > >
> > > > >
> > > > > 2018-03-09 10:15 GMT-08:00 Naveen Swamy :
> > > > > > Hi Guys,
> > > > > >
> > > > > > I am working on MXNet Scala Inference APIs
> > > > > >  along with
> > another
> > > > > > contributor Roshani. A while back I noticed that we haven't been
> > > > > publishing
> > > > > > the scala package to Maven for a while now(last one being
> v0.11.1a
> > > > under
> > > > > > the dmlc namespace).
> > > > > > Currently users have to build the package manually and then use
> it,
> > > > this
> > > > > > h

RE: call for contributions to next MXNet release

[Zhao, Patric]

Hi Steffen,

We'd like the MKL-DNN backend can be included in the next release.

We (Intel engineers)  and Zheng-Da (AWS engineer)  can work on it.

Could you help add the item in the table?

Thanks,

--Patric


> -Original Message-
> From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> Sent: Friday, March 9, 2018 9:29 PM
> To: dev@mxnet.incubator.apache.org
> Subject: call for contributions to next MXNet release
> 
> Hi - I would like to propose the next MXNet release.
> Initial draft content is listed at MXNet wiki
>  r+next+MXNet+Release>.
> I would like to call to all contributors to add features you are working on 
> and
> would like to see included in the release. Suggested code freeze is March
> 30th  with target release by mid April.
> 
> Anirudh Subramanian and Chris Olivier have volunteered to co-manage the
> release.
> 
> Mark your date: we are planing public meetup on Apr 24th in Seattle.
> Details to follow.
> 
> Regards,
> Steffen

Re: call for contributions to next MXNet release

[Marco de Abreu]

Hello Patric,

please be aware of the fact that there are still a lot of disabled tests,
open MKLDNN issues, broken features and flaky tests that have not been
addressed yet. We agreed on merging MKLDNN for the time being to allow
broad testing and that we will revisit the state a bit before the next
release. At the moment, I'd not be in favour of having MKLDNN being part of
it.

Best regards,
Marco

Zhao, Patric  schrieb am Sa., 10. März 2018, 02:30:

> Hi Steffen,
>
> We'd like the MKL-DNN backend can be included in the next release.
>
> We (Intel engineers)  and Zheng-Da (AWS engineer)  can work on it.
>
> Could you help add the item in the table?
>
> Thanks,
>
> --Patric
>
>
> > -Original Message-
> > From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> > Sent: Friday, March 9, 2018 9:29 PM
> > To: dev@mxnet.incubator.apache.org
> > Subject: call for contributions to next MXNet release
> >
> > Hi - I would like to propose the next MXNet release.
> > Initial draft content is listed at MXNet wiki
> >  > r+next+MXNet+Release>.
> > I would like to call to all contributors to add features you are working
> on and
> > would like to see included in the release. Suggested code freeze is March
> > 30th  with target release by mid April.
> >
> > Anirudh Subramanian and Chris Olivier have volunteered to co-manage the
> > release.
> >
> > Mark your date: we are planing public meetup on Apr 24th in Seattle.
> > Details to follow.
> >
> > Regards,
> > Steffen
>

Re: Publishing Scala Package/namespace change

[Chris Olivier]

user: “I’m having a problem with Scala API”.
dev: “What version?”
user: “1.4”
dev: “Is that Scala API version or MXNET version?”
user: “uhh... scala version, maybe?”
dev: “ok, which mxnet version?”
user: “i don’t know, how do I find that out?”
dev : “ummm go look in blah blah directory and tell me libmxnet.so file
date”
user: “nevermind, I’ll just use Caffe”


On Fri, Mar 9, 2018 at 4:43 PM Marco de Abreu 
wrote:

> My proposal was to have separate versioning but same release cycles -
> having separate cycles was just an idea.
>
> So how would we approach this problem in future if there's a big
> improvement on one of our language bindings? Like I said previously, it
> might be fine for this case, but we will run into the same issues for the
> next time and it's not a great user experience if we just do this
> "silently", provide no transitional solution and just replace an existing
> API with a new improved one.
>
> I would like to have a solution for this problem before we move on.
>
> -Marco
>
>
>
> On Sat, Mar 10, 2018 at 12:06 AM, Nan Zhu  wrote:
>
> > I think last time we postpone it because the release is a minor version
> >
> > but actually such a change is actually affordable for a jump from 1.1 -
> 1.2
> >
> > -1 on separate versions (not following apache rules)
> >
> >
> > On Fri, Mar 9, 2018 at 2:38 PM, Chris Olivier 
> > wrote:
> >
> > > IMHO, I don't think having separate versions and releases for the scala
> > API
> > > will work for the following reasons:
> > >
> > >- Scala API is not its own Apache project, so we don't really have a
> > >mechanism to release it separately and not the manpower of
> volunteers
> > to
> > >maintain all the BS that goes along with releases
> > >- We operate under the assumption (which is fair, I think) that the
> > >Scala API is only compatible with the exact C API for which it was
> > built
> > >against.  Since the Scala API ships with its own libmxnet.so, and
> > >libmxnet.so is "stable" only at release boundaries, then it would be
> > > risky
> > >to pick arbitrary points in the Scala evolution as "new versions"
> and
> > > birth
> > >them into the World with just whatever mxnet commit was at the top
> at
> > > that
> > >time
> > >
> > > I am also in favor of changing the namespace ASAP before the massive
> > > adoption of the Scala API that's about to happen, because it'll be way
> > > harder to do later.
> > >
> > > -Chris
> > >
> > >
> > > On Fri, Mar 9, 2018 at 2:09 PM, Naveen Swamy 
> wrote:
> > >
> > > > MXNet Scala APis already generate a MXNet Core Scala package based
> off
> > > the
> > > > cpp backend already. I think customers who are building from source
> > would
> > > > love to get Maven package given that it takes so much pain.
> > > >
> > > > Are you suggesting we take MXNet-Scala APIs into a separate release
> > > cycle,
> > > > it is possible and can start with this one but It would not make a
> lot
> > of
> > > > sense to start MXNet-Scala 1.0 depending on MXNet 1.0(cpp). This
> won't
> > > very
> > > > different from breaking backward compatibility when we release a new
> > > > package.
> > > >
> > > > IMO managing separate release cycles for different language bindings
> > > could
> > > > turn into a lot of work for the community unnecessarily especially
> > since
> > > > they are closely
> > > >
> > > > On Fri, Mar 9, 2018 at 1:56 PM, Marco de Abreu <
> > > > marco.g.ab...@googlemail.com
> > > > > wrote:
> > > >
> > > > > While it has never been published, there have still been releases
> > under
> > > > > Apache and - as you mentioned - customers that build off the
> source.
> > > This
> > > > > would cause compatibility issues.
> > > > >
> > > > > In general I actively support the idea of enhancing the Scala
> > package,
> > > > but
> > > > > I think that we have to solve another problem first. At the moment,
> > all
> > > > > APIs are bound to the MXNet core versioning and vica versa.
> > > > >
> > > > > In my opinion, we should first separate the APIs from the MXNet
> core,
> > > > start
> > > > > versioning them separately and then make changes like these. While
> it
> > > > would
> > > > > be possible (although not right) to make an exception here, we
> still
> > > > don't
> > > > > solve the root problem and we are going to run into the same issues
> > > with
> > > > > the next big API update.
> > > > >
> > > > > Just to mention another example: our team got a request to rewrite
> > the
> > > > Cpp
> > > > > package, but we actually would not be able to merge it into MXNet
> > since
> > > > it
> > > > > breaks the existing Cpp package API - means we would need a major
> > > version
> > > > > increase.
> > > > >
> > > > > We really should solve this problem once and for all, giving back a
> > lot
> > > > of
> > > > > freedom and reducing overhead in the long term.
> > > > >
> > > > > -Marco
> > > > >
> > > > > YiZhi Liu  schrieb am Fr., 9. März 2018,
> 22:44:
> > > > >
> > > > > > +1 fo

Re: Following semantic versioning

[Can Balioglu]

We found a rather "hacky" way to dynamically inject our operators and data 
iterators during runtime. I can share the details if you are interested. We 
also plan to send out a design proposal for a proper extensibility framework 
similar to TensorFlow's plugin API.

-Can

On Wed, Mar 7, 2018, at 00:28, Eric Xie wrote:
> One potential problem is with libstdc++. Specifically with vectors. Our 
> operator interface uses vectors, which breaks when core lib and 
> extension are compiled with different compiler version
> 
> On 2018/03/06 22:45:16, Chris Olivier  wrote: 
> > The static init of your module should register your operator just as it
> > does for the operators in mxnet (NNVM_REGISTER_OP).  While I haven't done
> > it personally, I see no reason why it wouldn't work like any other operator
> > at that point.
> > 
> > On Tue, Mar 6, 2018 at 1:28 PM, Barber, Christopher <
> > christopher.bar...@analog.com> wrote:
> > 
> > > There are two separate issues: how to compile against the MXNet source and
> > > how to dynamically load external dynamic libraries. While it would be nice
> > > to have all necessary headers in the same place, it doesn't really matter
> > > that much if people building extensions have to have access to the entire
> > > MXNet source tree. The real issue is whether you support the ability to
> > > dynamically load such extensions.
> > >
> > > - Christopher
> > >
> > > On 3/6/18, 4:12 PM, "Chris Olivier"  wrote:
> > >
> > > This was discussed in the past and so far, it seems possible for Unix
> > > builds, although it’s going to be messy since the compile would
> > > generally
> > > need access to all a large portion of the headers in the source tree,
> > > since
> > > the “things needed to create your own op†aren’t necessarily all
> > > contained
> > > in the include/mxnet directory.
> > >
> > > On Tue, Mar 6, 2018 at 11:40 AM kellen sunderland <
> > > kellen.sunderl...@gmail.com> wrote:
> > >
> > > > Could we actually just define a mechanism so the libs could register
> > > their
> > > > ops at runtime?  No linking required?
> > > >
> > > > On Tue, Mar 6, 2018, 8:36 PM Pedro Larroy <
> > > pedro.larroy.li...@gmail.com>
> > > > wrote:
> > > >
> > > > > This is a good point. What additional blockers would there be for
> > > linking
> > > > > against a user provided library with custom operators?
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 6, 2018 at 5:16 PM, Barber, Christopher <
> > > > > christopher.bar...@analog.com> wrote:
> > > > >
> > > > > > To avoid this kind of problem, you really need to support
> > > features that
> > > > > > allow MXNet to be extended without having to resort to forking.
> > > There
> > > > is
> > > > > > currently no way to add C++ custom operators without forking,
> > > and no
> > > > way
> > > > > to
> > > > > > share such operators across projects. This creates a perverse
> > > incentive
> > > > > to
> > > > > > try to get changes that may not belong into the main product.
> > > > > >
> > > > > > On 3/5/18, 6:26 PM, "Marco de Abreu" <
> > > marco.g.ab...@googlemail.com>
> > > > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > we recently had a few cases in which it has been attempted
> > > to add
> > > > new
> > > > > > functionality to old branches because a customer of 
> > > somebodys
> > > > > $DAY_JOB
> > > > > > requested it and was unable to switch to the latest release
> > > or that
> > > > > > certain
> > > > > > feature did not make it into the release. This lead to quite
> > > a lot
> > > > of
> > > > > > discussions and there was no clear standing within the
> > > community.
> > > > > >
> > > > > > Just to cite semantic versioning:
> > > > > >
> > > > > >1. MAJOR version when you make incompatible API changes,
> > > > > >2. MINOR version when you add functionality in a
> > > > > > backwards-compatible
> > > > > >manner, and
> > > > > >3. PATCH version when you make backwards-compatible bug
> > > fixes.
> > > > > >
> > > > > >
> > > > > > We as a community agreed on following this system and I
> > > think we
> > > > > should
> > > > > > either stick to it or drop it entirely - exceptions to
> > > SemVer are
> > > > > > usually
> > > > > > discouraged. While I see that adding functionality might be
> > > a minor
> > > > > > thing,
> > > > > > I don't think that we should educate our users into
> > > expecting us to
> > > > > > backport new features. The development happening on the
> > > Apache
> > > > MXNet
> > > > > > repository should not be influenced by something like this;
> > > > > especially
> > > > > > considering that whoever supports that customer 

Re: Following semantic versioning

[Chris Olivier]

Great!! Yes, please share!

On Fri, Mar 9, 2018 at 7:29 PM Can Balioglu  wrote:

> We found a rather "hacky" way to dynamically inject our operators and data
> iterators during runtime. I can share the details if you are interested. We
> also plan to send out a design proposal for a proper extensibility
> framework similar to TensorFlow's plugin API.
>
> -Can
>
> On Wed, Mar 7, 2018, at 00:28, Eric Xie wrote:
> > One potential problem is with libstdc++. Specifically with vectors. Our
> > operator interface uses vectors, which breaks when core lib and
> > extension are compiled with different compiler version
> >
> > On 2018/03/06 22:45:16, Chris Olivier  wrote:
> > > The static init of your module should register your operator just as it
> > > does for the operators in mxnet (NNVM_REGISTER_OP).  While I haven't
> done
> > > it personally, I see no reason why it wouldn't work like any other
> operator
> > > at that point.
> > >
> > > On Tue, Mar 6, 2018 at 1:28 PM, Barber, Christopher <
> > > christopher.bar...@analog.com> wrote:
> > >
> > > > There are two separate issues: how to compile against the MXNet
> source and
> > > > how to dynamically load external dynamic libraries. While it would
> be nice
> > > > to have all necessary headers in the same place, it doesn't really
> matter
> > > > that much if people building extensions have to have access to the
> entire
> > > > MXNet source tree. The real issue is whether you support the ability
> to
> > > > dynamically load such extensions.
> > > >
> > > > - Christopher
> > > >
> > > > On 3/6/18, 4:12 PM, "Chris Olivier" 
> wrote:
> > > >
> > > > This was discussed in the past and so far, it seems possible for
> Unix
> > > > builds, although it’s going to be messy since the compile would
> > > > generally
> > > > need access to all a large portion of the headers in the source
> tree,
> > > > since
> > > > the “things needed to create your own op†aren’t
> necessarily all
> > > > contained
> > > > in the include/mxnet directory.
> > > >
> > > > On Tue, Mar 6, 2018 at 11:40 AM kellen sunderland <
> > > > kellen.sunderl...@gmail.com> wrote:
> > > >
> > > > > Could we actually just define a mechanism so the libs could
> register
> > > > their
> > > > > ops at runtime?  No linking required?
> > > > >
> > > > > On Tue, Mar 6, 2018, 8:36 PM Pedro Larroy <
> > > > pedro.larroy.li...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is a good point. What additional blockers would there
> be for
> > > > linking
> > > > > > against a user provided library with custom operators?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 6, 2018 at 5:16 PM, Barber, Christopher <
> > > > > > christopher.bar...@analog.com> wrote:
> > > > > >
> > > > > > > To avoid this kind of problem, you really need to support
> > > > features that
> > > > > > > allow MXNet to be extended without having to resort to
> forking.
> > > > There
> > > > > is
> > > > > > > currently no way to add C++ custom operators without
> forking,
> > > > and no
> > > > > way
> > > > > > to
> > > > > > > share such operators across projects. This creates a
> perverse
> > > > incentive
> > > > > > to
> > > > > > > try to get changes that may not belong into the main
> product.
> > > > > > >
> > > > > > > On 3/5/18, 6:26 PM, "Marco de Abreu" <
> > > > marco.g.ab...@googlemail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > we recently had a few cases in which it has been
> attempted
> > > > to add
> > > > > new
> > > > > > > functionality to old branches because a customer of
> somebodys
> > > > > > $DAY_JOB
> > > > > > > requested it and was unable to switch to the latest
> release
> > > > or that
> > > > > > > certain
> > > > > > > feature did not make it into the release. This lead to
> quite
> > > > a lot
> > > > > of
> > > > > > > discussions and there was no clear standing within the
> > > > community.
> > > > > > >
> > > > > > > Just to cite semantic versioning:
> > > > > > >
> > > > > > >1. MAJOR version when you make incompatible API
> changes,
> > > > > > >2. MINOR version when you add functionality in a
> > > > > > > backwards-compatible
> > > > > > >manner, and
> > > > > > >3. PATCH version when you make backwards-compatible
> bug
> > > > fixes.
> > > > > > >
> > > > > > >
> > > > > > > We as a community agreed on following this system and I
> > > > think we
> > > > > > should
> > > > > > > either stick to it or drop it entirely - exceptions to
> > > > SemVer are
> > > > > > > usually
> > > > > > > discouraged. While I see that adding functionality
> might be
> > > > a minor
> > > > > > > thing,
> > > > > > > I don't t