Design Proposal: SVRG Optimization in MXNet Python Module

[Stephanie Yuan]

Hi MXNet dev community,

My name is Stephanie Yuan and it's great to join the MXNet dev family!  I'm
proposing a new design doc for implementing SVRG optimization technique in
MXNet Python Module.

*Problem Description: *
SVRG optimization is a technique that complements SGD, which was first
proposed in the paper  Accelerating Stochastic Gradient Descent using
Predicative Variance Reduction

in
2013.  It has provable guarantees for strongly convex functions and
converges much faster than SGD. An initial set of experiments using
YearPredictionMSD dataset has been conducted and yields promising results,
which is one of the motivations for this proposal.

*Expected Deliverables:*
The goal is to implement a MXNet Python Module that implements SVRG
optimization technique.

Detailed implementation approaches and Benchmark results can be found in
the Confluence design doc

.

Please let me know if you have any questions! Thank you very much for your
time and considerations!

Cheers,
Stephanie Yuan

Re: Seattle meetup

[Steffen Rochel]

Links to slides and recording area available below and on
https://cwiki.apache.org/confluence/display/MXNET/Seattle
Thanks again to Thom and Chenguang.

Enjoy!

   - Chenguang Wang: GluonNLP - a deep learning toolkit for Natural
   Language Processing - slides
   

and recording
   

   - Thom Lane: Handwriting OCR with MXNet on AWS SageMaker - slides
    and
   recording
   


On Thu, Aug 9, 2018 at 8:15 AM Steffen Rochel 
wrote:

> We will attempt to record and upload to cwiki.
> Steffen
>
> On Sat, Aug 4, 2018 at 5:21 AM Pedro Larroy 
> wrote:
>
>> Nice! Will this be broadcasted via Chime or other medium?
>>
>> On Wed, Aug 1, 2018 at 5:55 PM Steffen Rochel 
>> wrote:
>>
>> > I just updated the agenda for the meetup in Seattle on August 16th 2018.
>> > https://www.meetup.com/Apache-MXNet-Seattle-meetup/events/253104242/
>> > We will have two tutorials:
>> > (1) GluonNLP - a deep learning toolkit for Natural Language Processing
>> > (2) Handwriting OCR with MXNet on AWS SageMaker
>> >
>> > Please spread the word and hope to see you in Seattle. The plan is to
>> > record and publish the meetup.
>> > Please let me know suggestions and ideas you have for future topics.
>> >
>> > Regards,
>> > Steffen
>> >
>> > On Tue, Jul 31, 2018 at 2:21 PM Marco de Abreu
>> >  wrote:
>> >
>> > > Hi,
>> > >
>> > > I just found this Meetup invite:
>> > > https://www.meetup.com/Apache-MXNet-Seattle-meetup/events/253104242/
>> > >
>> > > Best regards,
>> > > Marco
>> > >
>> >
>>
>

Proposal: Apache MXNet user group meetings

[Davydenko, Denis]

Hello, Apache MXNet community,

 

I would like to submit for your consideration updated and improved suggestion 
for Apache MXNet User Groups meetings: [1]. Main goals of this initiative are:

- make it easy: users and developers should be able to understand what is Group 
Meeting and how to participate in it in under 60 seconds

- make it real: users and developers will have access to Apache MXNet 
committers and contributors live and can discuss their topics in person

- make it scalable: it should be easy for Apache MXNet community members to 
join initiative and host more User Groups meetings on their own schedule

- make it fun: there should be minimal process to run User Groups meetings

 

Please note this wiki page’s purpose is to introduce and describe initiative 
for Apache MXNet community and seek improvements/changes suggestions. User 
facing document will be separate and will only contain short paragraph on what 
is User Group meetings are and how to participate (see above: user should be 
able to understand it in under 60 seconds).

 

There were couple quite successful pilot sessions hosted by group of Apache 
MXNet developers in Berlin. Users clearly likes this channel of communication 
and this initiative is an effort to streamline it for bigger user participation.

 

Also, we are looking to add small reference to User Groups meetings into Apache 
MXNet documentation to bring user awareness up.

 

[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87299820

 

--

Thanks,

Denis

Re: Release plan - MXNET 1.3

[Roshani Nagmote]

Thanks Patric for reviewing the notes. Updated the doc with MKL-DNN points
you mentioned accordingly.

Regards,
Roshani

On Tue, Aug 21, 2018 at 8:03 PM Zhao, Patric  wrote:

> Hi Roshani,
>
> Good notes :)
>
> Several items about the performance and MKL-DNN in the below, please help
> take a review.
>
> @Da, Alex, if anything about MKL-DNN is missed, feel free to add.
>
> *Performance improvement
> +Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on CPU
> https://github.com/apache/incubator-mxnet/pull/3
> +Performance improvement for Batch Dot on CPU from mshadow
> https://github.com/dmlc/mshadow/pull/342
> -Fix the topk regression issue (#12197)
> This is the bugfix rather than performance improvements
>
>
> *MKL-DNN
> More functionality supports:
> +Support more activation functions, "sigmoid", "tanh", "softrelu"
> https://github.com/apache/incubator-mxnet/pull/10336
>
> Debugging functionality:
> +Result check
> https://github.com/apache/incubator-mxnet/pull/12069
> +Backend switch
> https://github.com/apache/incubator-mxnet/pull/12058
>
> Thanks,
>
> --Patric
>
> > -Original Message-
> > From: Roshani Nagmote [mailto:roshaninagmo...@gmail.com]
> > Sent: Wednesday, August 22, 2018 1:53 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: Release plan - MXNET 1.3
> >
> > Hi,
> >
> > Thank you everyone for helping to clear release blockers. CI tests were
> failing
> > so we delayed RC by some time. But now the tests are passing and we are
> > ready to cut the release branch.
> >
> > I have drafted release notes here:
> > https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28in
> > cubating%29+1.3.0+Release+Notes
> >
> >
> > Please take a look and update if I have missed anything. I will be
> cutting
> > RC0 tomorrow.
> >
> > Thanks,
> > Roshani
> >
> > On Thu, Aug 16, 2018 at 2:28 PM Roshani Nagmote
> > 
> > wrote:
> >
> > > Sure will do. thanks.
> > >
> > > -Roshani
> > >
> > > On Thu, Aug 16, 2018 at 11:53 AM Afrooze, Sina 
> > wrote:
> > >
> > >> Hi Roshani - Can you please make sure that this fix (which is already
> > >> merged to master) is also merged to the stable branch for 1.3.0:
> > >> https://github.com/apache/incubator-mxnet/pull/11493 - Thanks, Sina
> > >>
> > >>
> > >> On 8/16/18, 10:51 AM, "Roshani Nagmote"
> > 
> > >> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> Release status:
> > >>
> > >> Currently, for release 1.3.0 there are a couple of issues open
> > >> which needs
> > >> to be resolved before cutting RC.
> > >>
> > >> The current date we are looking at for cutting RC0 is
> 08/17(Friday).
> > >>
> > >>
> > >>
> > >> Open issues which need to be looked at before cutting RC:
> > >>
> > >>1. Topk regression issue
> > >> -
> > >> #12202 PR
> > >>with fix  >
> > >>2. Excessive memory allocation issue
> > >> -
> > >> #12184 PR
> > >>with fix  >
> > >>3. Test_io.test_csvIter breaks on CentOS
> > >> -
> > >> #12189 PR
> > >>with fix
> > >> 
> > >>
> > >>
> > >>
> > >> @committers, could you please help review these PRs and get them
> > >> merged?
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Roshani
> > >>
> > >> On Tue, Aug 14, 2018 at 12:46 PM Roshani Nagmote <
> > >> roshaninagmo...@gmail.com>
> > >> wrote:
> > >>
> > >> > Talked to the person who ran resnet50 benchmarks offline. Build
> > >> flag was
> > >> > not properly set so there was a difference in performance
> > >> numbers observed.
> > >> > There is no issue caught and he was able to get the same
> results as
> > >> > mentioned here https://mxnet.incubator.apache.org/faq/perf.html
> > >> > <
> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results>
> > >> >
> > >> > We are good here.
> > >> >
> > >> > Thanks,
> > >> > Roshani
> > >> >
> > >> > On Mon, Aug 13, 2018 at 4:08 PM Roshani Nagmote <
> > >> roshaninagmo...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hi Dom,
> > >> >>
> > >> >> I verified resnet50 run on MXNet master branch. Checked on
> > >> single gpu
> > >> >> machine. Numbers match. I didn't see any performance
> degradation.
> > >> >>
> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results
> > >> >>
> > >> >> Can you please give me more details on the instance type and
> > >> script you
> > >> >> ran exactly so that I can try to reproduce it again?
> > >> >>
> > >> >> Thanks,
> > >> >> Roshani
> > >> >>
> > >> >>
> > >> >> On Mon, Aug 13, 2018 at 12:31 PM Roshani Nagmote <
> > >> >> roshaninagmo.

Re: Proposal: Apache MXNet user group meetings

[Marco de Abreu]

Thanks for your proposal, Denis. This sounds great!

I especially like that we are making it easy for other groups to host these
sessions. That way, everybody can offer support in the schedule and for the
fields they feel most comfortable with. This makes it possible to directly
hear about the pain points of our developers and then immediately transfer
them into actionable items.

Best regards
Marco

Davydenko, Denis  schrieb am Mi., 22. Aug.
2018, 20:04:

> Hello, Apache MXNet community,
>
>
>
> I would like to submit for your consideration updated and improved
> suggestion for Apache MXNet User Groups meetings: [1]. Main goals of this
> initiative are:
>
> - make it easy: users and developers should be able to understand what is
> Group Meeting and how to participate in it in under 60 seconds
>
> - make it real: users and developers will have access to Apache MXNet
> committers and contributors live and can discuss their topics in person
>
> - make it scalable: it should be easy for Apache MXNet community members
> to join initiative and host more User Groups meetings on their own schedule
>
> - make it fun: there should be minimal process to run User Groups meetings
>
>
>
> Please note this wiki page’s purpose is to introduce and describe
> initiative for Apache MXNet community and seek improvements/changes
> suggestions. User facing document will be separate and will only contain
> short paragraph on what is User Group meetings are and how to participate
> (see above: user should be able to understand it in under 60 seconds).
>
>
>
> There were couple quite successful pilot sessions hosted by group of
> Apache MXNet developers in Berlin. Users clearly likes this channel of
> communication and this initiative is an effort to streamline it for bigger
> user participation.
>
>
>
> Also, we are looking to add small reference to User Groups meetings into
> Apache MXNet documentation to bring user awareness up.
>
>
>
> [1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87299820
>
>
>
> --
>
> Thanks,
>
> Denis
>
>

Re: Proposal: Apache MXNet user group meetings

[Chris Olivier]

The wiki lists 2111 Univ Ave, East Palo Alto, which is a body shop and/or
pharmacy across the street from the Amazon building, which is at 2100 Univ
Ave.

On Wed, Aug 22, 2018 at 3:10 PM Marco de Abreu
 wrote:

> Thanks for your proposal, Denis. This sounds great!
>
> I especially like that we are making it easy for other groups to host these
> sessions. That way, everybody can offer support in the schedule and for the
> fields they feel most comfortable with. This makes it possible to directly
> hear about the pain points of our developers and then immediately transfer
> them into actionable items.
>
> Best regards
> Marco
>
> Davydenko, Denis  schrieb am Mi., 22. Aug.
> 2018, 20:04:
>
> > Hello, Apache MXNet community,
> >
> >
> >
> > I would like to submit for your consideration updated and improved
> > suggestion for Apache MXNet User Groups meetings: [1]. Main goals of this
> > initiative are:
> >
> > - make it easy: users and developers should be able to understand what is
> > Group Meeting and how to participate in it in under 60 seconds
> >
> > - make it real: users and developers will have access to Apache MXNet
> > committers and contributors live and can discuss their topics in person
> >
> > - make it scalable: it should be easy for Apache MXNet community members
> > to join initiative and host more User Groups meetings on their own
> schedule
> >
> > - make it fun: there should be minimal process to run User Groups
> meetings
> >
> >
> >
> > Please note this wiki page’s purpose is to introduce and describe
> > initiative for Apache MXNet community and seek improvements/changes
> > suggestions. User facing document will be separate and will only contain
> > short paragraph on what is User Group meetings are and how to participate
> > (see above: user should be able to understand it in under 60 seconds).
> >
> >
> >
> > There were couple quite successful pilot sessions hosted by group of
> > Apache MXNet developers in Berlin. Users clearly likes this channel of
> > communication and this initiative is an effort to streamline it for
> bigger
> > user participation.
> >
> >
> >
> > Also, we are looking to add small reference to User Groups meetings into
> > Apache MXNet documentation to bring user awareness up.
> >
> >
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87299820
> >
> >
> >
> > --
> >
> > Thanks,
> >
> > Denis
> >
> >
>

Re: Proposal: Apache MXNet user group meetings

[Denis Davydenko]

Good catch, Chris! Will update it tomorrow. 

--
Thanks,
Denis

On Aug 22, 2018, at 7:57 PM, Chris Olivier  wrote:

The wiki lists 2111 Univ Ave, East Palo Alto, which is a body shop and/or
pharmacy across the street from the Amazon building, which is at 2100 Univ
Ave.

On Wed, Aug 22, 2018 at 3:10 PM Marco de Abreu
 wrote:

> Thanks for your proposal, Denis. This sounds great!
> 
> I especially like that we are making it easy for other groups to host these
> sessions. That way, everybody can offer support in the schedule and for the
> fields they feel most comfortable with. This makes it possible to directly
> hear about the pain points of our developers and then immediately transfer
> them into actionable items.
> 
> Best regards
> Marco
> 
> Davydenko, Denis  schrieb am Mi., 22. Aug.
> 2018, 20:04:
> 
>> Hello, Apache MXNet community,
>> 
>> 
>> 
>> I would like to submit for your consideration updated and improved
>> suggestion for Apache MXNet User Groups meetings: [1]. Main goals of this
>> initiative are:
>> 
>> - make it easy: users and developers should be able to understand what is
>> Group Meeting and how to participate in it in under 60 seconds
>> 
>> - make it real: users and developers will have access to Apache MXNet
>> committers and contributors live and can discuss their topics in person
>> 
>> - make it scalable: it should be easy for Apache MXNet community members
>> to join initiative and host more User Groups meetings on their own
> schedule
>> 
>> - make it fun: there should be minimal process to run User Groups
> meetings
>> 
>> 
>> 
>> Please note this wiki page’s purpose is to introduce and describe
>> initiative for Apache MXNet community and seek improvements/changes
>> suggestions. User facing document will be separate and will only contain
>> short paragraph on what is User Group meetings are and how to participate
>> (see above: user should be able to understand it in under 60 seconds).
>> 
>> 
>> 
>> There were couple quite successful pilot sessions hosted by group of
>> Apache MXNet developers in Berlin. Users clearly likes this channel of
>> communication and this initiative is an effort to streamline it for
> bigger
>> user participation.
>> 
>> 
>> 
>> Also, we are looking to add small reference to User Groups meetings into
>> Apache MXNet documentation to bring user awareness up.
>> 
>> 
>> 
>> [1]:
>> 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87299820
>> 
>> 
>> 
>> --
>> 
>> Thanks,
>> 
>> Denis
>> 
>> 
>