Re: Proposal to release the Examples for Scala in V1.3
Hi Marco, Currently all Scala examples I have changed contains test coverage on CI (in the example/src/test/). It's been temporarily disabled because of the memory leaks. I would also like to raise a discussion on a general standard for testing examples, this is what Scala side doing: 1. For small training examples we can finish them and check the dev accuracy 2. For big training examples, we do 5-10 Epoch to check it is runnable 3. For trivial examples (like neural style), we check it is runnable 4. For inference examples and classification specifically, we run with several test cases to measure if it can get the result correctly. Thanks, Qing On 7/12/18, 11:00 AM, "Marco de Abreu" wrote: Hi Qing, thanks a lot for your efforts around the Scala examples and to assist us getting the Scala API to a broader user base! This is a great user-facing approach! The following might not be in the current course of how we handle examples in Python, but let me elaborate: I have seen a lot of problems in the last months related to examples in Python in the last months that could have been avoided by testing them in integration tests. These errors included syntax errors, typos, wrong logic, outdated APIs or other more context-related errors. Thus, I'd like to strive for full test coverage of our examples in the long term. I know that we can't get this done within a single night, but since you are now reworking the way our examples work in Scala, I'd like to kindly request to assess whether it's possible to include testing of Scala examples in your plan right from the start. This will ensure that we have a good first impression with our users, increases our overall test coverage and also reduces the maintenance overhead we are having with examples breaking without us knowing. I think the memory leaks are adjacent problems and that examples are a good way to make them visible. Since they are already there, we shouldn't block ourselves from writing and publishing examples. Our users would encounter them anyways, but by making the examples, we can properly document and track them ahead of time. The big advantage right now would be that we could mark these things as known problems in our patch notes and thus our users would know that we are aware of them and that we are going to fix them by (hopefully) the following release. Best regards, Marco On Thu, Jul 12, 2018 at 8:41 PM YiZhi Liu wrote: > Agree to make incremental improvements. As long as the changes do not > change the current states of memory leak, but improve the documents > and demonstrate the way to use type-safe apis, I think it is fair to > merge into v1.3. > On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy wrote: > > > > Qing, its great that you are working on improving the Scala Examples with > > documentation and tests. > > Ideally, we shouldn't have any memory leaks..it erodes trust with our > > users, however I understand it can take significant time and debugging > > effort to resolve them. Given that these leaks already exist, may be > should > > call out on the documentation that this is a known issue and we will work > > on it. > > IMO for this case making incremental improvements to examples makes > sense. > > > > I would like to hear what others think. > > -Naveen > > > > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > I would like to propose the release plan for MXNet Scala examples in > V1.3. > > > Currently, the examples in the main repository are not maintained for a > > > long time. I have spent some time to use the improved Scala API (known > as > > > the .api) to improve the examples with documentation to run and add CI > > > tests. However, I am facing the difficulties to resolve memory leaks as > > > most of the examples did not contains memory leak fixes. It crashes > the CI > > > because of that. We can temporarily disable the CI test for some of the > > > examples and release them with 1.3 and enable them once we have a > better GC > > > in MXNet Scala. Here is a brief Pros and Cons for this release: > > > > > > Pros: > > > > > > * A good set of demos on how to use improved Training API > > > * Examples are fixed with full documentation to help to run them > > > > > > Cons: > > > > > > * Examples contains the memory leak issues may cause problems > > > > > > Please kindly share your ideas in this thread and I am really > appreciate > > > for your help. > > > > > > Thanks, > > > Qing > > > > > > > -- > Yizhi Liu > DMLC member > Amazon Web Services > Vancouver, Canada >
Re: Proposal to release the Examples for Scala in V1.3
Hi Qing, thanks a lot for your efforts around the Scala examples and to assist us getting the Scala API to a broader user base! This is a great user-facing approach! The following might not be in the current course of how we handle examples in Python, but let me elaborate: I have seen a lot of problems in the last months related to examples in Python in the last months that could have been avoided by testing them in integration tests. These errors included syntax errors, typos, wrong logic, outdated APIs or other more context-related errors. Thus, I'd like to strive for full test coverage of our examples in the long term. I know that we can't get this done within a single night, but since you are now reworking the way our examples work in Scala, I'd like to kindly request to assess whether it's possible to include testing of Scala examples in your plan right from the start. This will ensure that we have a good first impression with our users, increases our overall test coverage and also reduces the maintenance overhead we are having with examples breaking without us knowing. I think the memory leaks are adjacent problems and that examples are a good way to make them visible. Since they are already there, we shouldn't block ourselves from writing and publishing examples. Our users would encounter them anyways, but by making the examples, we can properly document and track them ahead of time. The big advantage right now would be that we could mark these things as known problems in our patch notes and thus our users would know that we are aware of them and that we are going to fix them by (hopefully) the following release. Best regards, Marco On Thu, Jul 12, 2018 at 8:41 PM YiZhi Liu wrote: > Agree to make incremental improvements. As long as the changes do not > change the current states of memory leak, but improve the documents > and demonstrate the way to use type-safe apis, I think it is fair to > merge into v1.3. > On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy wrote: > > > > Qing, its great that you are working on improving the Scala Examples with > > documentation and tests. > > Ideally, we shouldn't have any memory leaks..it erodes trust with our > > users, however I understand it can take significant time and debugging > > effort to resolve them. Given that these leaks already exist, may be > should > > call out on the documentation that this is a known issue and we will work > > on it. > > IMO for this case making incremental improvements to examples makes > sense. > > > > I would like to hear what others think. > > -Naveen > > > > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > I would like to propose the release plan for MXNet Scala examples in > V1.3. > > > Currently, the examples in the main repository are not maintained for a > > > long time. I have spent some time to use the improved Scala API (known > as > > > the .api) to improve the examples with documentation to run and add CI > > > tests. However, I am facing the difficulties to resolve memory leaks as > > > most of the examples did not contains memory leak fixes. It crashes > the CI > > > because of that. We can temporarily disable the CI test for some of the > > > examples and release them with 1.3 and enable them once we have a > better GC > > > in MXNet Scala. Here is a brief Pros and Cons for this release: > > > > > > Pros: > > > > > > * A good set of demos on how to use improved Training API > > > * Examples are fixed with full documentation to help to run them > > > > > > Cons: > > > > > > * Examples contains the memory leak issues may cause problems > > > > > > Please kindly share your ideas in this thread and I am really > appreciate > > > for your help. > > > > > > Thanks, > > > Qing > > > > > > > -- > Yizhi Liu > DMLC member > Amazon Web Services > Vancouver, Canada >
Re: Proposal to release the Examples for Scala in V1.3
Agree to make incremental improvements. As long as the changes do not change the current states of memory leak, but improve the documents and demonstrate the way to use type-safe apis, I think it is fair to merge into v1.3. On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy wrote: > > Qing, its great that you are working on improving the Scala Examples with > documentation and tests. > Ideally, we shouldn't have any memory leaks..it erodes trust with our > users, however I understand it can take significant time and debugging > effort to resolve them. Given that these leaks already exist, may be should > call out on the documentation that this is a known issue and we will work > on it. > IMO for this case making incremental improvements to examples makes sense. > > I would like to hear what others think. > -Naveen > > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan wrote: > > > Hi all, > > > > I would like to propose the release plan for MXNet Scala examples in V1.3. > > Currently, the examples in the main repository are not maintained for a > > long time. I have spent some time to use the improved Scala API (known as > > the .api) to improve the examples with documentation to run and add CI > > tests. However, I am facing the difficulties to resolve memory leaks as > > most of the examples did not contains memory leak fixes. It crashes the CI > > because of that. We can temporarily disable the CI test for some of the > > examples and release them with 1.3 and enable them once we have a better GC > > in MXNet Scala. Here is a brief Pros and Cons for this release: > > > > Pros: > > > > * A good set of demos on how to use improved Training API > > * Examples are fixed with full documentation to help to run them > > > > Cons: > > > > * Examples contains the memory leak issues may cause problems > > > > Please kindly share your ideas in this thread and I am really appreciate > > for your help. > > > > Thanks, > > Qing > > -- Yizhi Liu DMLC member Amazon Web Services Vancouver, Canada
Re: Proposal to release the Examples for Scala in V1.3
Qing, its great that you are working on improving the Scala Examples with documentation and tests. Ideally, we shouldn't have any memory leaks..it erodes trust with our users, however I understand it can take significant time and debugging effort to resolve them. Given that these leaks already exist, may be should call out on the documentation that this is a known issue and we will work on it. IMO for this case making incremental improvements to examples makes sense. I would like to hear what others think. -Naveen On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan wrote: > Hi all, > > I would like to propose the release plan for MXNet Scala examples in V1.3. > Currently, the examples in the main repository are not maintained for a > long time. I have spent some time to use the improved Scala API (known as > the .api) to improve the examples with documentation to run and add CI > tests. However, I am facing the difficulties to resolve memory leaks as > most of the examples did not contains memory leak fixes. It crashes the CI > because of that. We can temporarily disable the CI test for some of the > examples and release them with 1.3 and enable them once we have a better GC > in MXNet Scala. Here is a brief Pros and Cons for this release: > > Pros: > > * A good set of demos on how to use improved Training API > * Examples are fixed with full documentation to help to run them > > Cons: > > * Examples contains the memory leak issues may cause problems > > Please kindly share your ideas in this thread and I am really appreciate > for your help. > > Thanks, > Qing >
Proposal to release the Examples for Scala in V1.3
Hi all, I would like to propose the release plan for MXNet Scala examples in V1.3. Currently, the examples in the main repository are not maintained for a long time. I have spent some time to use the improved Scala API (known as the .api) to improve the examples with documentation to run and add CI tests. However, I am facing the difficulties to resolve memory leaks as most of the examples did not contains memory leak fixes. It crashes the CI because of that. We can temporarily disable the CI test for some of the examples and release them with 1.3 and enable them once we have a better GC in MXNet Scala. Here is a brief Pros and Cons for this release: Pros: * A good set of demos on how to use improved Training API * Examples are fixed with full documentation to help to run them Cons: * Examples contains the memory leak issues may cause problems Please kindly share your ideas in this thread and I am really appreciate for your help. Thanks, Qing
