Re: How should MXNet treat nan values?
Thanks to everyone who made their opinion known. So far the consensus is that any nan handling in MXNet should not affect performance, at least not by default. This still leaves the question open if we should aim for documenting the behavior of MXNet operators under presence of nan values. For example, should we include a sentence in the argmax and topk documentation? Should the 1.3 release notes note the changed behavior of topk? So far this has not been done. Instead any change of operator behavior with respect to nan values is treated as implementation change that is not worth noting to the user. As this can decrease user experience, I advocate for documenting the current behavior and possible future changes. In case there are no objections, is there any way to edit the changelog for the upcoming release?
Re: How should MXNet treat nan values?
+1 Performance should not be affected by default. Generally, I would treat nan as undefined behaviour and allow to explicitly enable checks for nan which will throw exceptions. Asmus Am Samstag, 21. Juli 2018, 21:31:05 MESZ hat Junru Shao Folgendes geschrieben: However, I am not 100% sure how much performance will be sacrificed if we stick to NumPy's approach which seems to check numeral exceptions on each step. I believe it will be great if we could make the default setting to be "no checking", and leave users an option to turn on these numeral exception checkings. On 2018/07/20 22:19:46, Leonard Lausen wrote: > Hello MXNet community, > > It seems that there is currently no agreed upon principle to handle > `nan` values in operators. This has led to inconsistencies between > operators and also to inconsistency over releases. Some operators ignore > nan values (eg. argmax), others treated it as maximum (e.g. topk up to > mxnet v1.2) or just return “undefined” output (e.g. topk starting with > mxnet v1.3). > > Initially the change in topk was reported as a bug > (https://github.com/apache/incubator-mxnet/issues/8510) as some users > relied on the behavior. However (and rightfully) @asmushetzel, who > contributed the improved topk operator for mxnet v1.3 pointed out that > the change did not break any documented behavior. > > To go forward, please share your opinion how MXNet should handle `nan` > values. Should we continue to treat the behavior as undefined and > possibly silently changing between releases? Should we define a > reasonable standard (e.g. follow numpy) and treat operators that deviate > as buggy? Should we just document how operators behave currently and > warn if the behavior changes? Something else? > > Please make your opinion known so above issue can be resolved/closed and > general guidelines can be defined for future contributions, following > whatever consensus emerges. > > Thanks! > Leonard >
Re: How should MXNet treat nan values?
If you behave like numpy for sparse, then things like dividing any sparse matrix by another sparse matrix will produce a dense matrix with a lot of NaNs in it wherever it encountered a “missing” value in both the source and destination positions of the sparse matrices (ie 0 divided by 0). If i remember correctly, this was the process at first (creating a dense matrix with lots of NaNs, which was equivalent to the same calculation with dense matrices in numpy) but then was replaced by just assuming that you didn’t want those NaNs and what you really wanted was a sparse result. On Sat, Jul 21, 2018 at 12:31 PM Junru Shao wrote: > However, I am not 100% sure how much performance will be sacrificed if we > stick to NumPy's approach which seems to check numeral exceptions on each > step. > > I believe it will be great if we could make the default setting to be "no > checking", and leave users an option to turn on these numeral exception > checkings. > > On 2018/07/20 22:19:46, Leonard Lausen wrote: > > Hello MXNet community, > > > > It seems that there is currently no agreed upon principle to handle > > `nan` values in operators. This has led to inconsistencies between > > operators and also to inconsistency over releases. Some operators ignore > > nan values (eg. argmax), others treated it as maximum (e.g. topk up to > > mxnet v1.2) or just return “undefined” output (e.g. topk starting with > > mxnet v1.3). > > > > Initially the change in topk was reported as a bug > > (https://github.com/apache/incubator-mxnet/issues/8510) as some users > > relied on the behavior. However (and rightfully) @asmushetzel, who > > contributed the improved topk operator for mxnet v1.3 pointed out that > > the change did not break any documented behavior. > > > > To go forward, please share your opinion how MXNet should handle `nan` > > values. Should we continue to treat the behavior as undefined and > > possibly silently changing between releases? Should we define a > > reasonable standard (e.g. follow numpy) and treat operators that deviate > > as buggy? Should we just document how operators behave currently and > > warn if the behavior changes? Something else? > > > > Please make your opinion known so above issue can be resolved/closed and > > general guidelines can be defined for future contributions, following > > whatever consensus emerges. > > > > Thanks! > > Leonard > > >
Re: How should MXNet treat nan values?
However, I am not 100% sure how much performance will be sacrificed if we stick to NumPy's approach which seems to check numeral exceptions on each step. I believe it will be great if we could make the default setting to be "no checking", and leave users an option to turn on these numeral exception checkings. On 2018/07/20 22:19:46, Leonard Lausen wrote: > Hello MXNet community, > > It seems that there is currently no agreed upon principle to handle > `nan` values in operators. This has led to inconsistencies between > operators and also to inconsistency over releases. Some operators ignore > nan values (eg. argmax), others treated it as maximum (e.g. topk up to > mxnet v1.2) or just return “undefined” output (e.g. topk starting with > mxnet v1.3). > > Initially the change in topk was reported as a bug > (https://github.com/apache/incubator-mxnet/issues/8510) as some users > relied on the behavior. However (and rightfully) @asmushetzel, who > contributed the improved topk operator for mxnet v1.3 pointed out that > the change did not break any documented behavior. > > To go forward, please share your opinion how MXNet should handle `nan` > values. Should we continue to treat the behavior as undefined and > possibly silently changing between releases? Should we define a > reasonable standard (e.g. follow numpy) and treat operators that deviate > as buggy? Should we just document how operators behave currently and > warn if the behavior changes? Something else? > > Please make your opinion known so above issue can be resolved/closed and > general guidelines can be defined for future contributions, following > whatever consensus emerges. > > Thanks! > Leonard >
Re: How should MXNet treat nan values?
I think it is worth discussing. NunPy has defined its own rules to handle with numeral exceptions, which makes a lot of sense to me. [Link](https://docs.scipy.org/doc/numpy/user/misc.html#how-numpy-handles-numerical-exceptions) On 2018/07/20 22:19:46, Leonard Lausen wrote: > Hello MXNet community, > > It seems that there is currently no agreed upon principle to handle > `nan` values in operators. This has led to inconsistencies between > operators and also to inconsistency over releases. Some operators ignore > nan values (eg. argmax), others treated it as maximum (e.g. topk up to > mxnet v1.2) or just return “undefined” output (e.g. topk starting with > mxnet v1.3). > > Initially the change in topk was reported as a bug > (https://github.com/apache/incubator-mxnet/issues/8510) as some users > relied on the behavior. However (and rightfully) @asmushetzel, who > contributed the improved topk operator for mxnet v1.3 pointed out that > the change did not break any documented behavior. > > To go forward, please share your opinion how MXNet should handle `nan` > values. Should we continue to treat the behavior as undefined and > possibly silently changing between releases? Should we define a > reasonable standard (e.g. follow numpy) and treat operators that deviate > as buggy? Should we just document how operators behave currently and > warn if the behavior changes? Something else? > > Please make your opinion known so above issue can be resolved/closed and > general guidelines can be defined for future contributions, following > whatever consensus emerges. > > Thanks! > Leonard >
How should MXNet treat nan values?
Hello MXNet community, It seems that there is currently no agreed upon principle to handle `nan` values in operators. This has led to inconsistencies between operators and also to inconsistency over releases. Some operators ignore nan values (eg. argmax), others treated it as maximum (e.g. topk up to mxnet v1.2) or just return “undefined” output (e.g. topk starting with mxnet v1.3). Initially the change in topk was reported as a bug (https://github.com/apache/incubator-mxnet/issues/8510) as some users relied on the behavior. However (and rightfully) @asmushetzel, who contributed the improved topk operator for mxnet v1.3 pointed out that the change did not break any documented behavior. To go forward, please share your opinion how MXNet should handle `nan` values. Should we continue to treat the behavior as undefined and possibly silently changing between releases? Should we define a reasonable standard (e.g. follow numpy) and treat operators that deviate as buggy? Should we just document how operators behave currently and warn if the behavior changes? Something else? Please make your opinion known so above issue can be resolved/closed and general guidelines can be defined for future contributions, following whatever consensus emerges. Thanks! Leonard