Re: Automatically translating Caffe code to MXNet

[Chris Olivier]

Great job!!!  Sounds awesome!



On Mon, Dec 4, 2017 at 4:30 PM, Indhu  wrote:

> One of the questions we frequently encounter in forums and emails is “I
> have caffe code that does this. How do I do the same thing in MXNet”. While
> it might be possible to go through the documentation and do the translation
> layer by layer manually, we thought a tool that automatically does it will
> be helpful for people who want to migrate from Caffe to MXNet. Here is the
> first version of the tool:
> https://github.com/apache/incubator-mxnet/tree/master/
> tools/caffe_translator.
> Thanks to Aaron (@aaronmarkham), Madan (@madjam), Pracheer (@pracheer) and
> Steffen (@srochel) for their help and feedback in building this.
>
> Note that this is different from the caffe converter that already exists (
> https://github.com/apache/incubator-mxnet/tree/master/
> tools/caffe_converter).
> Caffe converter converts Caffe Model to MXNet model so that user can run
> inference on MXNet or finetune on MXNet. The new translator on the other
> hand helps users to migrate their Caffe code to MXNet and continue
> development on MXNet. I think both tools are useful in different contexts.
>
> Caffe Translator can currently translate the layers mentioned in the README
>  tools/caffe_translator/README.md>.
> Obviously this is only a humble beginning and there is much more layers
> that can be supported. Apart from adding support for more layers, following
> are some things that can make the tool more useful:
>
> - Eliminate the CaffeDataIter
>  >
> dependency to translate data layers.
> - Generate code that can do distributed training. This is one of main
> reasons I might want to move my Caffe code to MXNet.
> - Make the tool accessible through a website so that user can just copy
> paste a Caffe code snippet in the website and get translated code instantly
> without downloading and running anything locally.
>
> Any help in these efforts is greatly appreciated. Also let me know if there
> is some other improvement that could be done to make the tool more useful.
>
> Thanks,
> Indu
>

Re: Note on when to Delete GitHub Comments

[Isabel Drost-Fromm]

Am 4. Dezember 2017 01:08:37 MEZ schrieb Hen :
>Basically a reminder that it should be a PMC decision, or if needing to
>move quickly a decision of the PMC Chair (mentors while a podling).

Remember that for provenance reasons all communication on github is mirrored on 
Apache servers in its own mailing list so you don't need to be afraid you loose 
information by accidentally clicking some delete button.

That's important to remember in particular if someone accidentally disclosed 
sensitive information e.g. as part of some bug report.

See here for more on that: http://apache.org/foundation/public-archives.html

See here for mailing list archives including your own (click through to a 
specific list to get to the search interface, you can search all the way 
through the entire history of the foundation's communication, powered by 
ponymail): 
https://lists.apache.org

If you'd like to improve ponymail, your patches are welcome. See here:
https://ponymail.incubator.apache.org


Isabel

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Automatically translating Caffe code to MXNet

[Indhu]

One of the questions we frequently encounter in forums and emails is “I
have caffe code that does this. How do I do the same thing in MXNet”. While
it might be possible to go through the documentation and do the translation
layer by layer manually, we thought a tool that automatically does it will
be helpful for people who want to migrate from Caffe to MXNet. Here is the
first version of the tool:
https://github.com/apache/incubator-mxnet/tree/master/tools/caffe_translator.
Thanks to Aaron (@aaronmarkham), Madan (@madjam), Pracheer (@pracheer) and
Steffen (@srochel) for their help and feedback in building this.

Note that this is different from the caffe converter that already exists (
https://github.com/apache/incubator-mxnet/tree/master/tools/caffe_converter).
Caffe converter converts Caffe Model to MXNet model so that user can run
inference on MXNet or finetune on MXNet. The new translator on the other
hand helps users to migrate their Caffe code to MXNet and continue
development on MXNet. I think both tools are useful in different contexts.

Caffe Translator can currently translate the layers mentioned in the README
.
Obviously this is only a humble beginning and there is much more layers
that can be supported. Apart from adding support for more layers, following
are some things that can make the tool more useful:

- Eliminate the CaffeDataIter

dependency to translate data layers.
- Generate code that can do distributed training. This is one of main
reasons I might want to move my Caffe code to MXNet.
- Make the tool accessible through a website so that user can just copy
paste a Caffe code snippet in the website and get translated code instantly
without downloading and running anything locally.

Any help in these efforts is greatly appreciated. Also let me know if there
is some other improvement that could be done to make the tool more useful.

Thanks,
Indu

[ANNOUNCE] Apache MXNet (incubating) 1.0.0 Release

[Chris Olivier]

Hello All,



The Apache MXNet (incubating) Community announces the availability of
Apache MXNet (incubating) 1.0.0!



Apache MXNet (incubating) is a deep learning framework designed for both
efficiency and flexibility. It allows you to mix symbolic and imperative
programming to maximize efficiency and productivity. We are excited to see
the broad base of users that are building production artificial
intelligence applications powered by neural network models developed and
trained with MXNet. With this release, MXNet simplifies training and
deploying deep learning models, and enables implementation of cutting-edge
performance enhancements.



A full list of the changes in this release can be found in the release
notes:

https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.0+Release+Notes




A Link to the Download is here:

https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.0.0




If you prefer to build from source and experiment with various compile-time
configuration options, use this link to get the instructions:

http://mxnet.incubator.apache.org/install/index.html



Or You can download and play with MXNet easily using one of the options
below:

   1. The Pip package can be found here: https://pypi.python.org/pypi/mxnet
   2. The Docker Images can be found here:
   https://hub.docker.com/r/mxnet/python/



The release tag used for the 1.0.0 release is:

https://github.com/apache/incubator-mxnet/tree/1.0.0



Some more MXNet Resources –

   1. Issues: https://github.com/apache/incubator-mxnet/issues
   2. Wiki: https://cwiki.apache.org/confluence/display/MXNET



If you want to learn more about MXNet visit
http://mxnet.incubator.apache.org/

Finally, you are welcome to join and also invite your friends to the
dynamic and growing MXNet community by subscribing to
dev@mxnet.incubator.apache.org



Thanks!

Apache MXNet (incubating) Team

___



DISCLAIMER:

Apache MXNet (incubating) is an effort undergoing incubation at The

Apache Software Foundation (ASF), sponsored by the name of Apache

Incubator PMC. Incubation is required of all newly accepted

projects until a further review indicates that the

infrastructure, communications, and decision making process have

stabilized in a manner consistent with other successful ASF

projects. While incubation status is not necessarily a reflection

of the completeness or stability of the code, it does indicate

that the project has yet to be fully endorsed by the ASF.

Re: [ANNOUNCE] Apache MXNet (incubating) 1.0.0 Release

[Spisak, Joseph]

Congrats
 

On 12/4/17, 4:01 PM, "Chris Olivier"  wrote:

Hello All,



The Apache MXNet (incubating) Community announces the availability of
Apache MXNet (incubating) 1.0.0!



Apache MXNet (incubating) is a deep learning framework designed for both
efficiency and flexibility. It allows you to mix symbolic and imperative
programming to maximize efficiency and productivity. We are excited to see
the broad base of users that are building production artificial
intelligence applications powered by neural network models developed and
trained with MXNet. With this release, MXNet simplifies training and
deploying deep learning models, and enables implementation of cutting-edge
performance enhancements.



A full list of the changes in this release can be found in the release
notes:


https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.0+Release+Notes




A Link to the Download is here:

https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.0.0




If you prefer to build from source and experiment with various compile-time
configuration options, use this link to get the instructions:

http://mxnet.incubator.apache.org/install/index.html



Or You can download and play with MXNet easily using one of the options
below:

   1. The Pip package can be found here: https://pypi.python.org/pypi/mxnet
   2. The Docker Images can be found here:
   https://hub.docker.com/r/mxnet/python/



The release tag used for the 1.0.0 release is:

https://github.com/apache/incubator-mxnet/tree/1.0.0



Some more MXNet Resources –

   1. Issues: https://github.com/apache/incubator-mxnet/issues
   2. Wiki: https://cwiki.apache.org/confluence/display/MXNET



If you want to learn more about MXNet visit
http://mxnet.incubator.apache.org/

Finally, you are welcome to join and also invite your friends to the
dynamic and growing MXNet community by subscribing to
dev@mxnet.incubator.apache.org



Thanks!

Apache MXNet (incubating) Team

___



DISCLAIMER:

Apache MXNet (incubating) is an effort undergoing incubation at The

Apache Software Foundation (ASF), sponsored by the name of Apache

Incubator PMC. Incubation is required of all newly accepted

projects until a further review indicates that the

infrastructure, communications, and decision making process have

stabilized in a manner consistent with other successful ASF

projects. While incubation status is not necessarily a reflection

of the completeness or stability of the code, it does indicate

that the project has yet to be fully endorsed by the ASF.

Re: [RFQ] Deprecate amalgamation

[Marco de Abreu]

Would you be able to us in touch with some customers who are using
amalgamation? That way we'd be able to gather some requirements and provide
them with a seamless replacement as part of our docker_multiarch
https://github.com/apache/incubator-mxnet/tree/master/docker_multiarch .

On Mon, Dec 4, 2017 at 10:54 PM, Lupesko, Hagay  wrote:

> JavaScript is not the only use case for Amalgamation - I’m familiar with a
> few users that use the amalgamation to build Android and iOS apps.
> If we take out Amalgamation, unless we provide target builds for these
> platforms, these users and use cases will be left out.
>
> Hagay
>
> On 11/21/17, 11:50, "Pedro Larroy"  wrote:
>
> Anybody against removing amalgamation then? emscripten build is
> already using CMake.
>
> On Tue, Nov 21, 2017 at 9:22 AM, Tianqi Chen 
> wrote:
> > Yes, you can call emscripten from CMake
> >
> > Tianqi
> >
> > On Mon, Nov 20, 2017 at 5:42 PM, Pedro Larroy <
> pedro.larroy.li...@gmail.com>
> > wrote:
> >
> >> I like the idea of amalgamation, I have used it in SQLite as it
> makes
> >> very easy to just drop one header file and one source file in
> another
> >> project to use the library.
> >>
> >> But SQLite is often used as a library embedded in platforms / other
> >> libraries.
> >>
> >> What's the use case of amalgamation in MXNet when we can build the
> >> binary library for all the platforms with MXNet's build system?  Who
> >> is using MXNet as an embedded library that can't use the shared
> >> library + headers or specific language bindings?
> >>
> >> Can't we call emscripten from CMake? I'm not familiar with our JS
> >> bindings, but I don't see why we can't compile for emscripten as for
> >> any other platform.
> >>
> >> Pedro.
> >>
> >> On Mon, Nov 20, 2017 at 11:59 PM, Tianqi Chen <
> tqc...@cs.washington.edu>
> >> wrote:
> >> > We could resort to a middle ground. Instead of having an
> amalgamation
> >> > script that generates a single file, simply have a file that
> includes
> >> > everything and compiles that one. Which should also work.
> >> >
> >> > The javascript port can likely be superseded with some form of
> support in
> >> > nnvm compiler, which transpires and generate likely more
> efficient code
> >> > than current version.  We can enable that feature now except that
> there
> >> is
> >> > no dedicated developer on it yet. We can talk about full
> deprecation
> >> after
> >> > this
> >> >
> >> >
> >> > Tianqi
> >> >
> >> > On Mon, Nov 20, 2017 at 2:47 PM, Pedro Larroy <
> >> pedro.larroy.li...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi all
> >> >>
> >> >> Given that we have working builds for ARM, Android, TX2 and the
> main
> >> >> architectures, and after considering how amalgamation is done. I
> would
> >> >> like to propose that we deprecate and remove amalgamation.
> >> >>
> >> >> I don't think the cost of maintaining this feature and how it's
> done
> >> >> justifies the ROI, given that we can now produce binary builds
> for
> >> >> embedded platforms in a comfortable way. It's also consuming
> build &
> >> >> test resources.
> >> >>
> >> >> We should strive to simplify our build system and development
> process.
> >> >>
> >> >> Pedro.
> >> >>
> >>
>
>
>
>

Re: [RFQ] Deprecate amalgamation

[Lupesko, Hagay]

JavaScript is not the only use case for Amalgamation - I’m familiar with a few 
users that use the amalgamation to build Android and iOS apps.
If we take out Amalgamation, unless we provide target builds for these 
platforms, these users and use cases will be left out.

Hagay

On 11/21/17, 11:50, "Pedro Larroy"  wrote:

Anybody against removing amalgamation then? emscripten build is
already using CMake.

On Tue, Nov 21, 2017 at 9:22 AM, Tianqi Chen  
wrote:
> Yes, you can call emscripten from CMake
>
> Tianqi
>
> On Mon, Nov 20, 2017 at 5:42 PM, Pedro Larroy 

> wrote:
>
>> I like the idea of amalgamation, I have used it in SQLite as it makes
>> very easy to just drop one header file and one source file in another
>> project to use the library.
>>
>> But SQLite is often used as a library embedded in platforms / other
>> libraries.
>>
>> What's the use case of amalgamation in MXNet when we can build the
>> binary library for all the platforms with MXNet's build system?  Who
>> is using MXNet as an embedded library that can't use the shared
>> library + headers or specific language bindings?
>>
>> Can't we call emscripten from CMake? I'm not familiar with our JS
>> bindings, but I don't see why we can't compile for emscripten as for
>> any other platform.
>>
>> Pedro.
>>
>> On Mon, Nov 20, 2017 at 11:59 PM, Tianqi Chen 
>> wrote:
>> > We could resort to a middle ground. Instead of having an amalgamation
>> > script that generates a single file, simply have a file that includes
>> > everything and compiles that one. Which should also work.
>> >
>> > The javascript port can likely be superseded with some form of support 
in
>> > nnvm compiler, which transpires and generate likely more efficient code
>> > than current version.  We can enable that feature now except that there
>> is
>> > no dedicated developer on it yet. We can talk about full deprecation
>> after
>> > this
>> >
>> >
>> > Tianqi
>> >
>> > On Mon, Nov 20, 2017 at 2:47 PM, Pedro Larroy <
>> pedro.larroy.li...@gmail.com>
>> > wrote:
>> >
>> >> Hi all
>> >>
>> >> Given that we have working builds for ARM, Android, TX2 and the main
>> >> architectures, and after considering how amalgamation is done. I would
>> >> like to propose that we deprecate and remove amalgamation.
>> >>
>> >> I don't think the cost of maintaining this feature and how it's done
>> >> justifies the ROI, given that we can now produce binary builds for
>> >> embedded platforms in a comfortable way. It's also consuming build &
>> >> test resources.
>> >>
>> >> We should strive to simplify our build system and development process.
>> >>
>> >> Pedro.
>> >>
>>

[BUILD FAILED] Branch master build 709

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/709/

[BUILD FAILED] Branch master build 710

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/710/

[BUILD FAILED] Branch master build 711

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/711/

Re: Protected master needs to be turned off

[Pedro Larroy]

Agree on both Mu and Tianqi’s points.


I think this would align well with the “nightly tests” narrative. We
can disable CI for trivial changes with a comment like (“skip ci”)
comment on the ghprb Jenkins plugin, and trust the nightly to catch
any problems introduced by trivial patches in an aggregated manner.

Pedro.

On Fri, Dec 1, 2017 at 6:49 PM, Tianqi Chen  wrote:
> I think we are still using CI to catch bugs. And we should take caution
> when merging something that CI did not catch up due to the response time.
> This do not contradict what comitter can do with their best judgement. For
> example, I would normally switch CI off and merge simple typo fixes. If the
> fix merged causes problem, you still get CI to report it maybe a few hours
> later.
>
> The bottom line is that comitter get these information as feedbacks and
> they use their best judgement when do so. Most of the time when unsure, I
> simply wait for CI to finish
>
> Tianqi
>
>
> On Fri, Dec 1, 2017 at 9:41 AM Mu Li  wrote:
>
>> Totally agree that CI is useful. Actually, I wrote the jenkinsfile and
>> setup the jenkins server before moving to apache server. I just mention
>> that we cannot rely on the CI test. It currently covers operator unittests
>> and regression test on several cnns. But the code coverage isn't great. If
>> a PR touches the core system, the best practice today is still code
>> reviewing. Otherwise, such as a PR is mainly about examples, the CI often
>> doesn't help so we just waste machine times.
>>
>> I think checking the exact code coverage is on the roadmap, but I don't
>> know if we have any progress on it.
>>
>> On Fri, Dec 1, 2017 at 6:19 AM, Pedro Larroy > >
>> wrote:
>>
>> > CI catches problems all the time. I don't think many of us can afford
>> > to build all the flavors and architectures in their laptops or
>> > workstations, so we have to rely on CI to catch all kinds of errors
>> > from compilation errors to bugs plus regressions, specially in a
>> > project which has so many build flavors.
>> >
>> > I have had this experience in big projects several times and I can
>> > tell you it's always the same.
>> >
>> > So from extensive software development experience I write that we will
>> > be able to develop and merge much faster once we have a reliable CI
>> > running in short cycles, any other approach or shortcuts is just
>> > accumulating technical debt for the future that somebody will have to
>> > cleanup and will slow down development. Is better to have a CI with a
>> > reduced scope working reliably than bypassing CI.
>> >
>> > This is irrespective of using dev to merge or unprotected master.
>> >
>> > We can't afford to have increased warnings, bugs creeping into the
>> > codebase going unnoticed, build system problems, performance
>> > regressions, etc. And we have to rely on a solid CI for this. If we
>> > are not ready for this, we should halt feature development or at least
>> > merging new features until we have a stable codebase and build system.
>> >
>>

Re: Note on when to Delete GitHub Comments

[Isabel Drost-Fromm]

Am 4. Dezember 2017 01:08:37 MEZ schrieb Hen :
>
>That gives me a chance to mention a PMC Chair's role.
>
>When MXNet graduates, one of the initial PMC members will be nominated
>to
>the board to be the chair. That chair is a VP/Officer of the 501(c)(3)
>and
>is the one who is ultimately responsible to the board for the project.
>Very
>occasionally a chair may have to do something without consulting with
>the
>PMC as a whole, and they're expected to get it right.

Some context on the duties of PMC chair:

https://www.apache.org/dev/pmc.html#chair

http://www.apache.org/foundation/governance/pmcs.html


http://www.apache.org/dev/pmc.html


Hope this helps though being a lot of text to go through.

Isabel



-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.