[GitHub] eric-haibin-lin commented on issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h
eric-haibin-lin commented on issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h URL: https://github.com/apache/incubator-mxnet/issues/8590#issuecomment-342734066 Is that a build failure? What's your MXNet version/commit id and output of `diagnose.py `? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on a change in pull request #8566: optimize broadcast
reminisce commented on a change in pull request #8566: optimize broadcast
URL: https://github.com/apache/incubator-mxnet/pull/8566#discussion_r149591356
##
File path: src/operator/mxnet_op.h
##
@@ -345,6 +394,13 @@ __global__ void mxnet_generic_kernel(int N, Args... args)
{
}
}
+template
+__global__ void mxnet_generic_kernel_ex(int N, Args... args) {
+ for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < N; i += blockDim.x *
gridDim.x) {
+OP::Map(i, 1, args...);
Review comment:
For GPU, can we also get the number of threads first, and set the
corresponding length here as you did for CPU? I think it will also accelerate
computation on GPU side because of less calls to the functions `unravel` and
`dot`, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] szha commented on a change in pull request #8512: gluon rnn refactor
szha commented on a change in pull request #8512: gluon rnn refactor URL: https://github.com/apache/incubator-mxnet/pull/8512#discussion_r149585223 ## File path: tests/python/unittest/test_gluon_rnn.py ## @@ -25,6 +25,7 @@ def test_rnn(): cell = gluon.rnn.RNNCell(100, prefix='rnn_') +print(cell) Review comment: Yes, this is for verifying whether cell `__repr__` works. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8512: gluon rnn refactor
eric-haibin-lin commented on a change in pull request #8512: gluon rnn refactor URL: https://github.com/apache/incubator-mxnet/pull/8512#discussion_r149585026 ## File path: tests/python/unittest/test_gluon_rnn.py ## @@ -25,6 +25,7 @@ def test_rnn(): cell = gluon.rnn.RNNCell(100, prefix='rnn_') +print(cell) Review comment: Are these prints intentional? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tz-hmc opened a new issue #8591: How do I make a siamese network with pretrained models (esp. keeping the weights the same?)
tz-hmc opened a new issue #8591: How do I make a siamese network with
pretrained models (esp. keeping the weights the same?)
URL: https://github.com/apache/incubator-mxnet/issues/8591
## Description
How do I ensure the weights are kept the same? Can I unpack the internal
layers somehow and set the weights of each to the same variable? My apologies,
I'm new to MXNet. Would really appreciate the help, thanks!
``
sym1, arg_params1, aux_params1 = mx.model.load_checkpoint('resnet-152', 0)
sym2, arg_params2, aux_params2 = mx.model.load_checkpoint('resnet-152', 0)
layer1 = sym1.get_internals()
layer2 = sym2.get_internals()
for i in range(len(layer1)): # will something like this work?
arg_params1[i] = arg_params2[i]
``
Relevant answers, but not specific enough to my particular problem:
https://github.com/apache/incubator-mxnet/issues/772siamese networks
https://github.com/apache/incubator-mxnet/issues/6791 extract layers as
variables
https://github.com/apache/incubator-mxnet/issues/557 set weights to be
same
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] 994088895 commented on issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h
994088895 commented on issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h URL: https://github.com/apache/incubator-mxnet/issues/8590#issuecomment-342721281 I have the error mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h, canyou tell me what should i do This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] 994088895 opened a new issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h
994088895 opened a new issue #8590: the error:mshadow::cuda::AddTakeGrad' : ambiguous call to overloaded function indexing_op.h URL: https://github.com/apache/incubator-mxnet/issues/8590 Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues. If the issue is non-technical, feel free to present the information in what you believe is the best form. ## Description (Brief description of the problem in no more than 2 sentences.) ## Environment info (Required) ``` What to do: 1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py 2. Run the script using `python diagnose.py` and paste its output here. ``` Package used (Python/R/Scala/Julia): (I'm using ...) For Scala user, please provide: 1. Java version: (`java -version`) 2. Maven version: (`mvn -version`) 3. Scala runtime if applicable: (`scala -version`) For R user, please provide R `sessionInfo()`: ## Build info (Required if built from source) Compiler (gcc/clang/mingw/visual studio): MXNet commit hash: (Paste the output of `git rev-parse HEAD` here.) Build config: (Paste the content of config.mk, or the build command.) ## Error Message: (Paste the complete error message, including stack trace.) ## Minimum reproducible example (If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.) ## Steps to reproduce (Paste the commands you ran that produced the error.) 1. 2. ## What have you tried to solve it? 1. 2. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ZiyueHuang commented on a change in pull request #8539: expose group2ctx to module
ZiyueHuang commented on a change in pull request #8539: expose group2ctx to
module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149582373
##
File path: tests/python/unittest/test_module.py
##
@@ -70,6 +70,63 @@ def test_module_input_grads():
assert np.all(c_grad == 3), c_grad
+def test_module_ctx_group():
+with mx.AttrScope(ctx_group='dev1'):
+a = mx.symbol.Variable('a')
+a = a * 2
+with mx.AttrScope(ctx_group='dev2'):
+b = mx.symbol.Variable('b')
+c = a + b
+shape = (2, 5)
+mod1 = mx.mod.Module(c, context=[mx.cpu(0)], data_names=['a', 'b'],
label_names=None,
+ group2ctxs=[{'dev1':mx.cpu(1),'dev2':mx.cpu(2)}])
+mod1.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod1.init_params()
+mod1.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod1.backward([mx.nd.ones(shape)])
+mod1_input_grads = mod1.get_input_grads()
+
+mod2 = mx.mod.Module(c, data_names=['a', 'b'], label_names=None)
+mod2.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod2.init_params()
+mod2.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod2.backward([mx.nd.ones(shape)])
+mod2_input_grads = mod2.get_input_grads()
+
+assert np.all(mod1_input_grads[0].asnumpy() ==
mod2_input_grads[0].asnumpy())
+assert np.all(mod1_input_grads[1].asnumpy() ==
mod2_input_grads[1].asnumpy())
+
+
+def test_bucket_module_ctx_group():
Review comment:
Just to verify that the change in bucket module is OK, i.e. the symbol can
successfully be binded to `group2ctxs`. Should I remove this test or `assert
self.binded`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ZiyueHuang commented on a change in pull request #8539: expose group2ctx to module
ZiyueHuang commented on a change in pull request #8539: expose group2ctx to
module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149582373
##
File path: tests/python/unittest/test_module.py
##
@@ -70,6 +70,63 @@ def test_module_input_grads():
assert np.all(c_grad == 3), c_grad
+def test_module_ctx_group():
+with mx.AttrScope(ctx_group='dev1'):
+a = mx.symbol.Variable('a')
+a = a * 2
+with mx.AttrScope(ctx_group='dev2'):
+b = mx.symbol.Variable('b')
+c = a + b
+shape = (2, 5)
+mod1 = mx.mod.Module(c, context=[mx.cpu(0)], data_names=['a', 'b'],
label_names=None,
+ group2ctxs=[{'dev1':mx.cpu(1),'dev2':mx.cpu(2)}])
+mod1.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod1.init_params()
+mod1.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod1.backward([mx.nd.ones(shape)])
+mod1_input_grads = mod1.get_input_grads()
+
+mod2 = mx.mod.Module(c, data_names=['a', 'b'], label_names=None)
+mod2.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod2.init_params()
+mod2.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod2.backward([mx.nd.ones(shape)])
+mod2_input_grads = mod2.get_input_grads()
+
+assert np.all(mod1_input_grads[0].asnumpy() ==
mod2_input_grads[0].asnumpy())
+assert np.all(mod1_input_grads[1].asnumpy() ==
mod2_input_grads[1].asnumpy())
+
+
+def test_bucket_module_ctx_group():
Review comment:
Just to verify that the change in bucket module is OK, i.e. the symbol can
successfully be binded to `group2ctxs`. Should I remove this test or `asset
self.binded`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ZiyueHuang commented on a change in pull request #8539: expose group2ctx to module
ZiyueHuang commented on a change in pull request #8539: expose group2ctx to
module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149582123
##
File path: python/mxnet/module/module.py
##
@@ -73,6 +75,11 @@ def __init__(self, symbol, data_names=('data',),
label_names=('softmax_label',),
assert len(work_load_list) == len(self._context)
self._work_load_list = work_load_list
+if group2ctxs is None:
Review comment:
Some user may use `DataExecutorGroup` directly. And there are such cases in
`test_shared_exec_group` in `test_module.py`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ZiyueHuang commented on a change in pull request #8539: expose group2ctx to module
ZiyueHuang commented on a change in pull request #8539: expose group2ctx to
module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149581557
##
File path: python/mxnet/module/module.py
##
@@ -59,10 +59,12 @@ class Module(BaseModule):
state_names : list of str
states are similar to data and label, but not provided by data
iterator.
Instead they are initialized to 0 and can be set by `set_states()`.
+group2ctxs : list of dict of str to context
+Default is `None`. Mapping the `ctx_group` attribute to the context
assignment.
"""
def __init__(self, symbol, data_names=('data',),
label_names=('softmax_label',),
logger=logging, context=ctx.cpu(), work_load_list=None,
- fixed_param_names=None, state_names=None):
+ fixed_param_names=None, state_names=None, group2ctxs=None):
Review comment:
I think it's better for `group2ctxs` to be in the same place with
`contexts`and `symbol`, i.e, in the constructor. Otherwise if an user use
`mod.fit`, he should pass `group2ctxs` to `fit`, I think this way seems kind of
weird.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ZiyueHuang commented on a change in pull request #8539: expose group2ctx to module
ZiyueHuang commented on a change in pull request #8539: expose group2ctx to
module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149581557
##
File path: python/mxnet/module/module.py
##
@@ -59,10 +59,12 @@ class Module(BaseModule):
state_names : list of str
states are similar to data and label, but not provided by data
iterator.
Instead they are initialized to 0 and can be set by `set_states()`.
+group2ctxs : list of dict of str to context
+Default is `None`. Mapping the `ctx_group` attribute to the context
assignment.
"""
def __init__(self, symbol, data_names=('data',),
label_names=('softmax_label',),
logger=logging, context=ctx.cpu(), work_load_list=None,
- fixed_param_names=None, state_names=None):
+ fixed_param_names=None, state_names=None, group2ctxs=None):
Review comment:
I think it's better for `group2ctxs` to be in the same place with `contexts`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8539: expose group2ctx to module
eric-haibin-lin commented on a change in pull request #8539: expose group2ctx
to module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149578575
##
File path: python/mxnet/module/module.py
##
@@ -73,6 +75,11 @@ def __init__(self, symbol, data_names=('data',),
label_names=('softmax_label',),
assert len(work_load_list) == len(self._context)
self._work_load_list = work_load_list
+if group2ctxs is None:
Review comment:
This seems redundant since you already checked `None` in DataExecutorGroup?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8539: expose group2ctx to module
eric-haibin-lin commented on a change in pull request #8539: expose group2ctx
to module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149578869
##
File path: python/mxnet/module/module.py
##
@@ -59,10 +59,12 @@ class Module(BaseModule):
state_names : list of str
states are similar to data and label, but not provided by data
iterator.
Instead they are initialized to 0 and can be set by `set_states()`.
+group2ctxs : list of dict of str to context
+Default is `None`. Mapping the `ctx_group` attribute to the context
assignment.
"""
def __init__(self, symbol, data_names=('data',),
label_names=('softmax_label',),
logger=logging, context=ctx.cpu(), work_load_list=None,
- fixed_param_names=None, state_names=None):
+ fixed_param_names=None, state_names=None, group2ctxs=None):
Review comment:
Hmmm. Is Module::bind a better place to put the `group2ctxs` arg instead of
the constructor, since it's only used during binding?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8539: expose group2ctx to module
eric-haibin-lin commented on a change in pull request #8539: expose group2ctx
to module
URL: https://github.com/apache/incubator-mxnet/pull/8539#discussion_r149578224
##
File path: tests/python/unittest/test_module.py
##
@@ -70,6 +70,63 @@ def test_module_input_grads():
assert np.all(c_grad == 3), c_grad
+def test_module_ctx_group():
+with mx.AttrScope(ctx_group='dev1'):
+a = mx.symbol.Variable('a')
+a = a * 2
+with mx.AttrScope(ctx_group='dev2'):
+b = mx.symbol.Variable('b')
+c = a + b
+shape = (2, 5)
+mod1 = mx.mod.Module(c, context=[mx.cpu(0)], data_names=['a', 'b'],
label_names=None,
+ group2ctxs=[{'dev1':mx.cpu(1),'dev2':mx.cpu(2)}])
+mod1.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod1.init_params()
+mod1.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod1.backward([mx.nd.ones(shape)])
+mod1_input_grads = mod1.get_input_grads()
+
+mod2 = mx.mod.Module(c, data_names=['a', 'b'], label_names=None)
+mod2.bind(data_shapes=[['a', shape], ['b', shape]], inputs_need_grad=True)
+mod2.init_params()
+mod2.forward(data_batch=mx.io.DataBatch(data=[mx.nd.ones(shape),
mx.nd.ones(shape)]), is_train=True)
+mod2.backward([mx.nd.ones(shape)])
+mod2_input_grads = mod2.get_input_grads()
+
+assert np.all(mod1_input_grads[0].asnumpy() ==
mod2_input_grads[0].asnumpy())
+assert np.all(mod1_input_grads[1].asnumpy() ==
mod2_input_grads[1].asnumpy())
+
+
+def test_bucket_module_ctx_group():
Review comment:
What does this test against? Anything to assert?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kevinthesun opened a new issue #8589: [Discussion] Data Pipeline Intermediate Representation in MXNet/NNVM
kevinthesun opened a new issue #8589: [Discussion] Data Pipeline Intermediate Representation in MXNet/NNVM URL: https://github.com/apache/incubator-mxnet/issues/8589 Tensorflow has a transform package https://github.com/tensorflow/transform which is capable of export a data preprocessing pipeline to a tensorflow graph, which can be incorporated into network graph. This package provides a neat way to manage data pipeline together with network graph and eliminates the gap of data preprocessing for training and inference(especially for serving application). Also I think we can get some performance improvement by using computation graph for data process rather than imperative processing for large data stream? Currently in MXNet, if I want to do the similar thing, I need to pack the code(most time python script) directly with network graph files. This method has some issues: 1. Potential security issue. If I wrote the processing codes and I am the only person use it, it's fine. However, if someone else wants to reuse it in their application, they need to check the code to make sure there is no security issue. It is not quite portable for reusing. 2. It is bind to specific language. Usually it's easier to develop deep learning application using python, but if my production environment doesn't have python environment, I need to either setup python environment or rewrite this script with the language supported by my production environment. Any thought about supporting data pipeline IR in MXNet/NNVM? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhreshold commented on issue #8562: Number of output from the mx.sym.SoftmaxOutput
zhreshold commented on issue #8562: Number of output from the mx.sym.SoftmaxOutput URL: https://github.com/apache/incubator-mxnet/issues/8562#issuecomment-342693812 what you need: ``` ... fc1_act = xxx fc2 = mx.symbol.FullyConnected(data=fc1_act, num_hidden=2) lenet = mx.sym.SoftmaxOutput(data=fc2,name='softmax') ... ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhreshold commented on issue #8573: /usr/bin/ld: cannot find -lgtest ?
zhreshold commented on issue #8573: /usr/bin/ld: cannot find -lgtest ? URL: https://github.com/apache/incubator-mxnet/issues/8573#issuecomment-342693279 To build from source, please follow this guide: http://mxnet.incubator.apache.org/install/index.html This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ZiqianXie commented on issue #8585: API document needs to be updated
ZiqianXie commented on issue #8585: API document needs to be updated URL: https://github.com/apache/incubator-mxnet/issues/8585#issuecomment-342691956 Sorry, I just realize that my version of mxnet is 0.11.0. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ZiqianXie closed issue #8585: API document needs to be updated
ZiqianXie closed issue #8585: API document needs to be updated URL: https://github.com/apache/incubator-mxnet/issues/8585 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] solin319 commented on issue #8584: Equivalent parameters for lr_mult and decay_mult of convolution in caffe in mxnet
solin319 commented on issue #8584: Equivalent parameters for lr_mult and decay_mult of convolution in caffe in mxnet URL: https://github.com/apache/incubator-mxnet/issues/8584#issuecomment-342691070 You can use set_lr_mult, set_wd_mult in class Optimizer. http://mxnet.incubator.apache.org/api/python/optimization/optimization.html This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #8378: upgrade MKL
szha commented on issue #8378: upgrade MKL URL: https://github.com/apache/incubator-mxnet/pull/8378#issuecomment-339418566 Thanks, @ykim362. @eric-haibin-lin @piiswrong shall we merge this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zihaolucky commented on issue #8583: Improve custom cross-entropy operator example
zihaolucky commented on issue #8583: Improve custom cross-entropy operator example URL: https://github.com/apache/incubator-mxnet/issues/8583#issuecomment-342690181 Add a performance test against NDArry and Numpy array would be great. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] solin319 commented on issue #8583: Improve custom cross-entropy operator example
solin319 commented on issue #8583: Improve custom cross-entropy operator example URL: https://github.com/apache/incubator-mxnet/issues/8583#issuecomment-342689116 I pull a new request in #8588. The test result were equal between use np.array and mx.NDArray. The first test result are as below. ``` use np.array D:[[ 0.69253981 0.29041654 0.23231442 0.5502885 ] [ 0.71507961 0.42464432 0.97114891 0.68113315] [ 0.48131326 0.39427516 0.34631446 0.72446871] [ 0.4398008 0.06848434 0.40008336 0.73323548] [ 0.18884189 0.18194272 0.53092033 0.53119105] [ 0.63171291 0.84244317 0.71996623 0.60880303]] L:[[ 0. 0. 1. 1.] [ 0. 0. 1. 0.] [ 1. 0. 0. 1.] [ 0. 1. 0. 1.] [ 0. 0. 0. 0.] [ 1. 0. 0. 1.]] out:[[-1.17940652 -0.34307581 -1.45965922 -0.59731078] [-1.25554216 -0.55276525 -0.02927442 -1.14297867] [-0.73123485 -0.50132793 -0.42512736 -0.32231531] [-0.57946116 -2.68113565 -0.51096296 -0.310287 ] [-0.2092911 -0.20082173 -0.7569806 -0.75755793] [-0.45931864 -1.84796298 -1.27284169 -0.49625883]] use mx.nd.array D:[[ 0.69253981 0.29041654 0.23231442 0.5502885 ] [ 0.71507961 0.42464432 0.97114891 0.68113315] [ 0.48131326 0.39427516 0.34631446 0.72446871] [ 0.4398008 0.06848434 0.40008336 0.73323548] [ 0.18884189 0.18194272 0.53092033 0.53119105] [ 0.63171291 0.84244317 0.71996623 0.60880303]] L:[[ 0. 0. 1. 1.] [ 0. 0. 1. 0.] [ 1. 0. 0. 1.] [ 0. 1. 0. 1.] [ 0. 0. 0. 0.] [ 1. 0. 0. 1.]] out:[[-1.17940652 -0.34307581 -1.45965922 -0.59731078] [-1.25554216 -0.55276525 -0.02927442 -1.14297867] [-0.73123485 -0.50132793 -0.42512736 -0.32231531] [-0.57946116 -2.68113565 -0.51096296 -0.310287 ] [-0.2092911 -0.20082173 -0.7569806 -0.75755793] [-0.45931864 -1.84796298 -1.27284169 -0.49625883]] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhreshold commented on issue #8563: Inconsistent Python calls, Python output different from C++ output
zhreshold commented on issue #8563: Inconsistent Python calls, Python output different from C++ output URL: https://github.com/apache/incubator-mxnet/issues/8563#issuecomment-342681055 Okay, this turned out to be not a bug, it's the dropout behavior. Turn off is_train mode will resolve the issue. ``` mod.forward(batch, is_train=False) ``` In c++, you are in prediction forward perfectly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] solin319 opened a new pull request #8588: implement cross-entropy operator with NDArray
solin319 opened a new pull request #8588: implement cross-entropy operator with NDArray URL: https://github.com/apache/incubator-mxnet/pull/8588 #8583 In the recommender example, reimplement cross-entropy operator with NDArray. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ShownX commented on issue #8559: How to use Prelu in gluon?
ShownX commented on issue #8559: How to use Prelu in gluon? URL: https://github.com/apache/incubator-mxnet/issues/8559#issuecomment-342687345 How? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] anirudh2290 opened a new issue #8587: Issue with CSVIter for small datasets
anirudh2290 opened a new issue #8587: Issue with CSVIter for small datasets URL: https://github.com/apache/incubator-mxnet/issues/8587 Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues. If the issue is non-technical, feel free to present the information in what you believe is the best form. ## Description CSVIter crashes when used with small datasets. ## Environment info (Required) Deep Learning AMI Package used (Python/R/Scala/Julia): Python MXNet commit hash: 90ba62a9eae9061d3763d594a552a367e550fe4f Build config: ## Error Message: [[[ 1. 2.] [ 3. 4.]]] [[[ 3. 4.] [ 5. 6.]]] [[[ 6. 7.] [ 8. 9.]]] [[[ 9. 10.] [ 11. 12.]]] [02:06:01] /home/ubuntu/sparse_support/mxnet/dmlc-core/include/dmlc/./logging.h:308: [02:06:01] src/io/iter_csv.cc:124: Check failed: row.length == shape.Size() (1 vs. 4) The data size in CSV do not match size of shape: specified shape=[2,2], the csv row-length=1 Stack trace returned 10 entries: [bt] (0) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3f) [0x7ff1ce214f7f] [bt] (1) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2io7CSVIter7AsTBlobERKN4dmlc3RowIjEERKN4nnvm6TShapeE+0x14a) [0x7ff1cfde0dc2] [bt] (2) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2io7CSVIter4NextEv+0x25e) [0x7ff1cfde08a2] [bt] (3) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet2io11BatchLoader4NextEv+0xa1) [0x7ff1cfd9b2cf] [bt] (4) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZZN5mxnet2io14PrefetcherIter4InitERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_ESaISA_EEENKUlPPNS_9DataBatchEE_clESH_+0x50) [0x7ff1cfd9796a] [bt] (5) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFbPPN5mxnet9DataBatchEEZNS0_2io14PrefetcherIter4InitERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESE_ESaISF_EEEUlS3_E_E9_M_invokeERKSt9_Any_dataOS3_+0x37) [0x7ff1cfda83a3] [bt] (6) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNKSt8functionIFbPPN5mxnet9DataBatchEEEclES3_+0x49) [0x7ff1cfda8811] [bt] (7) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZZN4dmlc12ThreadedIterIN5mxnet9DataBatchEE4InitESt8functionIFbPPS2_EES4_IFvvEEENKUlvE_clEv+0x311) [0x7ff1cfda3457] [bt] (8) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt12_Bind_simpleIFZN4dmlc12ThreadedIterIN5mxnet9DataBatchEE4InitESt8functionIFbPPS3_EES5_IFvvEEEUlvE_vEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE+0x28) [0x7ff1cfdbab06] [bt] (9) /home/ubuntu/sparse_support/mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt12_Bind_simpleIFZN4dmlc12ThreadedIterIN5mxnet9DataBatchEE4InitESt8functionIFbPPS3_EES5_IFvvEEEUlvE_vEEclEv+0x2c) [0x7ff1cfdb8602] terminate called after throwing an instance of 'dmlc::Error' what(): [02:06:01] src/io/iter_csv.cc:124: Check failed: row.length == shape.Size() (1 vs. 4) The data size in CSV do not match size of shape: specified shape=[2,2], the csv row-length=1 ## Minimum reproducible example ``` import mxnet as mx data_path = 'x.t' data_train = None data_train = mx.io.CSVIter(data_csv=data_path, data_shape=(2,2), batch_size=1) for batch in iter(data_train): print data_train.getdata().asnumpy() ``` Here is how my x.t looks like: >>> cat x.t 1,2,3,4 3,4,5,6 6,7,8,9 9,10,11,12 ## Steps to reproduce 1. Create a file x.t with the contents above. 2. Run the python script above ## What have you tried to solve it? 1. Increased the data size. This issue is reproduced only for small datasets. For data with more than 64 rows I was not able to reproduce the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] lyblsgo opened a new issue #8586: Word-level language modeling RNN, Training on multiple GPUs
lyblsgo opened a new issue #8586: Word-level language modeling RNN, Training on multiple GPUs URL: https://github.com/apache/incubator-mxnet/issues/8586 How to train Word-level-language-modeling-RNN on multiple GPUs, does anyone have try this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch master updated: [sparse] slice for csr on two dimensions, cpu implementation (#8331)
This is an automated email from the ASF dual-hosted git repository.
jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new bf2336c [sparse] slice for csr on two dimensions, cpu implementation
(#8331)
bf2336c is described below
commit bf2336c1f54e98a10793296ea09929624cddb4c6
Author: Ziyue Huang
AuthorDate: Tue Nov 7 19:56:47 2017 -0600
[sparse] slice for csr on two dimensions, cpu implementation (#8331)
* slice axis for csr (cpu impl)
* fix indice bug and use kernel launch
* small fix
* misc updates to address comments
* fix type
* csr slice
* unittest
* fix lint
* address comments
* return csr zeros before kernel launch if nnz=0
* fix
---
src/operator/tensor/matrix_op-inl.h | 193 ++-
src/operator/tensor/matrix_op.cc | 6 +-
tests/python/unittest/test_sparse_ndarray.py | 49 ---
3 files changed, 188 insertions(+), 60 deletions(-)
diff --git a/src/operator/tensor/matrix_op-inl.h
b/src/operator/tensor/matrix_op-inl.h
index 42a95b8..e012efe 100644
--- a/src/operator/tensor/matrix_op-inl.h
+++ b/src/operator/tensor/matrix_op-inl.h
@@ -443,8 +443,7 @@ inline bool SliceForwardInferStorageType(const
nnvm::NodeAttrs& attrs,
dispatch_mode, DispatchMode::kFCompute);
}
- if (!dispatched && in_stype == kCSRStorage && param.begin.ndim() <= 1 &&
- param.end.ndim() <= 1) {
+ if (!dispatched && in_stype == kCSRStorage) {
dispatched = storage_type_assign(&out_stype, kCSRStorage,
dispatch_mode, dispatch_ex);
}
@@ -551,48 +550,43 @@ void SliceCsrIndPtrImpl(const int begin, const int end,
RunContext ctx,
}
/*
- * Slice a CSR NDArray
+ * Slice a CSR NDArray for first dimension
* Only implemented for CPU
*/
template
-void SliceCsrImpl(const SliceParam ¶m, const OpContext& ctx,
- const NDArray &in, OpReqType req, const NDArray &out) {
+void SliceDimOneCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
using namespace mshadow;
using namespace mxnet_op;
using namespace csr;
- CHECK((std::is_same::value)) << "Slice for CSR input only
implemented for CPU";
- if (req == kNullOp) return;
- CHECK_NE(req, kAddTo) << "kAddTo for Slice on CSR input is not supported";
- CHECK_NE(req, kWriteInplace) << "kWriteInplace for Slice on CSR input is not
supported";
- const TShape ishape = in.shape();
- int begin = *param.begin[0];
- if (begin < 0) begin += ishape[0];
- int end = *param.end[0];
- if (end < 0) end += ishape[0];
- int indptr_len = end - begin + 1;
+ CHECK((std::is_same::value)) << "SliceDimOneCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0];
+ nnvm::dim_t end_row = end[0];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
- if (!in.storage_initialized()) {
-out.set_aux_shape(kIndPtr, Shape1(0));
-return;
- }
// assume idx indptr share the same type
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
-auto in_indptr = in.aux_data(kIndPtr).dptr();
-auto out_indptr = out.aux_data(kIndPtr).dptr();
-SliceCsrIndPtrImpl(begin, end, ctx.run_ctx, in_indptr,
out_indptr);
+RType* in_indptr = in.aux_data(kIndPtr).dptr();
+RType* out_indptr = out.aux_data(kIndPtr).dptr();
+SliceCsrIndPtrImpl(begin_row, end_row, ctx.run_ctx,
in_indptr, out_indptr);
// retrieve nnz (CPU implementation)
int nnz = out_indptr[indptr_len - 1];
+// return csr zeros if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
+ return;
+}
// copy indices and values
out.CheckAndAllocAuxData(kIdx, Shape1(nnz));
out.CheckAndAllocData(Shape1(nnz));
-auto in_idx = in.aux_data(kIdx).dptr();
-auto out_idx = out.aux_data(kIdx).dptr();
-auto in_data = in.data().dptr();
-auto out_data = out.data().dptr();
-int offset = in_indptr[begin];
+IType* in_idx = in.aux_data(kIdx).dptr();
+IType* out_idx = out.aux_data(kIdx).dptr();
+DType* in_data = in.data().dptr();
+DType* out_data = out.data().dptr();
+int offset = in_indptr[begin_row];
// this is also a CPU-only implementation
memcpy(out_idx, in_idx + offset, nnz * sizeof(IType));
memcpy(out_data, in_data + offset, nnz * sizeof(DType));
@@ -601,18 +595,151 @@ void SliceCsrImpl(const SliceParam ¶m, const
OpContext& ctx,
});
}
+/*!
+ * \brief slice a CSRNDA
[GitHub] piiswrong closed pull request #8331: [sparse] slice for csr on two dimensions, cpu implementation
piiswrong closed pull request #8331: [sparse] slice for csr on two dimensions,
cpu implementation
URL: https://github.com/apache/incubator-mxnet/pull/8331
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/operator/tensor/matrix_op-inl.h
b/src/operator/tensor/matrix_op-inl.h
index 42a95b88c3..e012efe25b 100644
--- a/src/operator/tensor/matrix_op-inl.h
+++ b/src/operator/tensor/matrix_op-inl.h
@@ -443,8 +443,7 @@ inline bool SliceForwardInferStorageType(const
nnvm::NodeAttrs& attrs,
dispatch_mode, DispatchMode::kFCompute);
}
- if (!dispatched && in_stype == kCSRStorage && param.begin.ndim() <= 1 &&
- param.end.ndim() <= 1) {
+ if (!dispatched && in_stype == kCSRStorage) {
dispatched = storage_type_assign(&out_stype, kCSRStorage,
dispatch_mode, dispatch_ex);
}
@@ -551,48 +550,43 @@ void SliceCsrIndPtrImpl(const int begin, const int end,
RunContext ctx,
}
/*
- * Slice a CSR NDArray
+ * Slice a CSR NDArray for first dimension
* Only implemented for CPU
*/
template
-void SliceCsrImpl(const SliceParam ¶m, const OpContext& ctx,
- const NDArray &in, OpReqType req, const NDArray &out) {
+void SliceDimOneCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
using namespace mshadow;
using namespace mxnet_op;
using namespace csr;
- CHECK((std::is_same::value)) << "Slice for CSR input only
implemented for CPU";
- if (req == kNullOp) return;
- CHECK_NE(req, kAddTo) << "kAddTo for Slice on CSR input is not supported";
- CHECK_NE(req, kWriteInplace) << "kWriteInplace for Slice on CSR input is not
supported";
- const TShape ishape = in.shape();
- int begin = *param.begin[0];
- if (begin < 0) begin += ishape[0];
- int end = *param.end[0];
- if (end < 0) end += ishape[0];
- int indptr_len = end - begin + 1;
+ CHECK((std::is_same::value)) << "SliceDimOneCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0];
+ nnvm::dim_t end_row = end[0];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
- if (!in.storage_initialized()) {
-out.set_aux_shape(kIndPtr, Shape1(0));
-return;
- }
// assume idx indptr share the same type
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
-auto in_indptr = in.aux_data(kIndPtr).dptr();
-auto out_indptr = out.aux_data(kIndPtr).dptr();
-SliceCsrIndPtrImpl(begin, end, ctx.run_ctx, in_indptr,
out_indptr);
+RType* in_indptr = in.aux_data(kIndPtr).dptr();
+RType* out_indptr = out.aux_data(kIndPtr).dptr();
+SliceCsrIndPtrImpl(begin_row, end_row, ctx.run_ctx,
in_indptr, out_indptr);
// retrieve nnz (CPU implementation)
int nnz = out_indptr[indptr_len - 1];
+// return csr zeros if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
+ return;
+}
// copy indices and values
out.CheckAndAllocAuxData(kIdx, Shape1(nnz));
out.CheckAndAllocData(Shape1(nnz));
-auto in_idx = in.aux_data(kIdx).dptr();
-auto out_idx = out.aux_data(kIdx).dptr();
-auto in_data = in.data().dptr();
-auto out_data = out.data().dptr();
-int offset = in_indptr[begin];
+IType* in_idx = in.aux_data(kIdx).dptr();
+IType* out_idx = out.aux_data(kIdx).dptr();
+DType* in_data = in.data().dptr();
+DType* out_data = out.data().dptr();
+int offset = in_indptr[begin_row];
// this is also a CPU-only implementation
memcpy(out_idx, in_idx + offset, nnz * sizeof(IType));
memcpy(out_data, in_data + offset, nnz * sizeof(DType));
@@ -601,18 +595,151 @@ void SliceCsrImpl(const SliceParam ¶m, const
OpContext& ctx,
});
}
+/*!
+ * \brief slice a CSRNDArray for two dimensions
+ */
+struct SliceDimTwoCsrAssign {
+ /*!
+ * \brief This function slices a CSRNDArray on axis one between begin_col
and end_col
+ * \param i loop index
+ * \param out_idx output csr ndarray column indices
+ * \param out_dataoutput csr ndarray data
+ * \param out_indptr output csr ndarray row index pointer
+ * \param in_idx input csr ndarray column indices
+ * \param in_data input csr ndarray data
+ * \param in_indptr input csr ndarray row index pointer
+ * \param begin_col begin column indice
+ * \param end_col end column indice
+ */
+ template
+ MSHADOW_XINLINE static void Map(int i,
+
[GitHub] ZiqianXie opened a new issue #8585: API document needs to be updated
ZiqianXie opened a new issue #8585: API document needs to be updated URL: https://github.com/apache/incubator-mxnet/issues/8585 [http://mxnet.incubator.apache.org/api/python/ndarray/random.html#module-mxnet.ndarray.random](Here) the API document says that the mxnet.ndarray.random is the random distribution generator module but in the current version it is in mxnet.random. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for csr on two dimensions, cpu implementation
eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for
csr on two dimensions, cpu implementation
URL: https://github.com/apache/incubator-mxnet/pull/8331#discussion_r149505376
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -601,18 +595,151 @@ void SliceCsrImpl(const SliceParam ¶m, const
OpContext& ctx,
});
}
+/*!
+ * \brief slice a CSRNDArray for two dimensions
+ */
+struct SliceDimTwoCsrAssign {
+ /*!
+ * \brief This function slices a CSRNDArray on axis one between begin_col
and end_col
+ * \param i loop index
+ * \param out_idx output csr ndarray column indices
+ * \param out_dataoutput csr ndarray data
+ * \param out_indptr output csr ndarray row index pointer
+ * \param in_idx input csr ndarray column indices
+ * \param in_data input csr ndarray data
+ * \param in_indptr input csr ndarray row index pointer
+ * \param begin_col begin column indice
+ * \param end_col end column indice
+ */
+ template
+ MSHADOW_XINLINE static void Map(int i,
+ IType* out_idx, DType* out_data,
+ const RType* out_indptr,
+ const IType* in_idx, const DType* in_data,
+ const RType* in_indptr,
+ const int begin_col, const int end_col) {
+RType ind = out_indptr[i];
+for (RType j = in_indptr[i]; j < in_indptr[i+1]; j++) {
+ // indices of CSRNDArray are in ascending order per row
+ if (in_idx[j] >= end_col) {
+break;
+ } else if (in_idx[j] >= begin_col) {
+out_idx[ind] = in_idx[j] - begin_col;
+out_data[ind] = in_data[j];
+ind++;
+ }
+}
+ }
+};
+
+/*
+ * Slice a CSR NDArray for two dimensions
+ * Only implemented for CPU
+ */
+template
+void SliceDimTwoCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
+ using namespace mshadow;
+ using namespace mxnet_op;
+ using namespace csr;
+ CHECK((std::is_same::value)) << "SliceDimTwoCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0], end_row = end[0];
+ nnvm::dim_t begin_col = begin[1], end_col = end[1];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
+ out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
+ // assume idx indptr share the same type
+ MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
+MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
+ MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
+RType *in_indptr = in.aux_data(kIndPtr).dptr();
+IType *in_idx = in.aux_data(kIdx).dptr();
+DType *in_data = in.data().dptr();
+// retrieve nnz (CPU implementation)
+RType *out_indptr = out.aux_data(kIndPtr).dptr();
+int nnz = 0;
+out_indptr[0] = 0;
+// loop through indptr array and corresponding indices to count for nnz
+for (nnvm::dim_t i = 0; i < indptr_len - 1; i++) {
+ out_indptr[i+1] = out_indptr[i];
+ for (RType j = in_indptr[i + begin_row];
+ j < in_indptr[i + begin_row + 1]; j++) {
+// indices of CSRNDArray are in ascending order per row
+if (in_idx[j] >= end_col) {
+ break;
+} else if (in_idx[j] >= begin_col) {
+ out_indptr[i+1]++;
+ nnz++;
+}
+ }
+}
+// returns zeros in csr format if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
Review comment:
Same comment here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for csr on two dimensions, cpu implementation
eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for
csr on two dimensions, cpu implementation
URL: https://github.com/apache/incubator-mxnet/pull/8331#discussion_r149505293
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -551,48 +550,43 @@ void SliceCsrIndPtrImpl(const int begin, const int end,
RunContext ctx,
}
/*
- * Slice a CSR NDArray
+ * Slice a CSR NDArray for first dimension
* Only implemented for CPU
*/
template
-void SliceCsrImpl(const SliceParam ¶m, const OpContext& ctx,
- const NDArray &in, OpReqType req, const NDArray &out) {
+void SliceDimOneCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
using namespace mshadow;
using namespace mxnet_op;
using namespace csr;
- CHECK((std::is_same::value)) << "Slice for CSR input only
implemented for CPU";
- if (req == kNullOp) return;
- CHECK_NE(req, kAddTo) << "kAddTo for Slice on CSR input is not supported";
- CHECK_NE(req, kWriteInplace) << "kWriteInplace for Slice on CSR input is not
supported";
- const TShape ishape = in.shape();
- int begin = *param.begin[0];
- if (begin < 0) begin += ishape[0];
- int end = *param.end[0];
- if (end < 0) end += ishape[0];
- int indptr_len = end - begin + 1;
+ CHECK((std::is_same::value)) << "SliceDimOneCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0];
+ nnvm::dim_t end_row = end[0];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
- if (!in.storage_initialized()) {
-out.set_aux_shape(kIndPtr, Shape1(0));
-return;
- }
// assume idx indptr share the same type
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
-auto in_indptr = in.aux_data(kIndPtr).dptr();
-auto out_indptr = out.aux_data(kIndPtr).dptr();
-SliceCsrIndPtrImpl(begin, end, ctx.run_ctx, in_indptr,
out_indptr);
+RType* in_indptr = in.aux_data(kIndPtr).dptr();
+RType* out_indptr = out.aux_data(kIndPtr).dptr();
+SliceCsrIndPtrImpl(begin_row, end_row, ctx.run_ctx,
in_indptr, out_indptr);
// retrieve nnz (CPU implementation)
int nnz = out_indptr[indptr_len - 1];
+// return csr zeros if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
Review comment:
According to https://github.com/apache/incubator-mxnet/pull/7935 setting the
aux shape is not sufficient to generate a valid CSR ndarray. Let's always call
`ZerosCsrImpl` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zhreshold commented on issue #8563: Inconsistent Python calls, Python output different from C++ output
zhreshold commented on issue #8563: Inconsistent Python calls, Python output different from C++ output URL: https://github.com/apache/incubator-mxnet/issues/8563#issuecomment-342681055 This is weird, so which variant of mxnet are you using? Say mxnet, mxnet-mkl, mxnet-cu80(90), mxnet-cu80(90)-mkl This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] BiranLi commented on a change in pull request #8373: distribute training in fp16
BiranLi commented on a change in pull request #8373: distribute training in fp16
URL: https://github.com/apache/incubator-mxnet/pull/8373#discussion_r149554775
##
File path: src/kvstore/kvstore_dist.h
##
@@ -45,12 +45,14 @@ namespace kvstore {
* it's the server node's job to control the data consistency among all
* workers. see details on \ref ServerHandle::Start
*/
+
+template
class KVStoreDist : public KVStoreLocal {
Review comment:
Sorry, I have make a mistake in kvstore optimizer codes. I was tortured for
several days with this error code. It can work well now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chowkamlee81 opened a new issue #8584: Equivalent parameters for lr_mult and decay_mult of convolution in caffe in mxnet
chowkamlee81 opened a new issue #8584: Equivalent parameters for lr_mult and
decay_mult of convolution in caffe in mxnet
URL: https://github.com/apache/incubator-mxnet/issues/8584
I want to convert PSPNET written in caffe to mxnet. In caffe under
convolution and batch normalization lr_mult and decay_mult parameters are there
sample prototxt below
layer {
name: "conv5_3_pool2_conv"
type: "Convolution"
bottom: "conv5_3_pool2"
top: "conv5_3_pool2_conv"
param {
lr_mult: 10
decay_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 1
stride: 1
weight_filler {
type: "msra"
}
bias_term: false
}
}
layer {
name: "conv5_3_pool2_conv/bn"
type: "BN"
bottom: "conv5_3_pool2_conv"
top: "conv5_3_pool2_conv"
param {
lr_mult: 10
decay_mult: 0
}
param {
lr_mult: 10
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
bn_param {
slope_filler {
type: "constant"
value: 1
}
bias_filler {
type: "constant"
value: 0
}
frozen: true
momentum: 0.95
}
}
I would like to have lr_mult and decay_mult for convolution and batch
normalization of caffe equivalent in mxnet. Kindly suggest
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ShownX commented on issue #8581: More examples ?
ShownX commented on issue #8581: More examples ? URL: https://github.com/apache/incubator-mxnet/issues/8581#issuecomment-342679374 How about here: https://github.com/apache/incubator-mxnet/tree/master/example/gluon/word_language_model This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] solin319 commented on issue #8583: Improve custom cross-entropy operator example
solin319 commented on issue #8583: Improve custom cross-entropy operator example URL: https://github.com/apache/incubator-mxnet/issues/8583#issuecomment-342678867 I can have a try to implement cross-entropy operator with NDArray. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhreshold commented on issue #8423: Re-implement segnet in MXnet
zhreshold commented on issue #8423: Re-implement segnet in MXnet URL: https://github.com/apache/incubator-mxnet/pull/8423#issuecomment-342675988 @solin319 Have you finished? Can you rebase to fix the CI? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha closed pull request #8516: Update mshadow to 2d7780c3f2eefe4453fa419862d1b2089bedb8d5
szha closed pull request #8516: Update mshadow to 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 URL: https://github.com/apache/incubator-mxnet/pull/8516 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/mshadow b/mshadow index cb5c9872b5..2d7780c3f2 16 --- a/mshadow +++ b/mshadow @@ -1 +1 @@ -Subproject commit cb5c9872b542220be5b99f3aca0e1ff56e31b028 +Subproject commit 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch master updated: Update mshadow to 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 (#8516)
This is an automated email from the ASF dual-hosted git repository. zhasheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new bbf8b3e Update mshadow to 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 (#8516) bbf8b3e is described below commit bbf8b3e8838ad6531a38de8087ac6055b92802ca Author: Pedro Larroy <928489+lar...@users.noreply.github.com> AuthorDate: Wed Nov 8 01:47:08 2017 +0100 Update mshadow to 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 (#8516) --- mshadow | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mshadow b/mshadow index cb5c987..2d7780c 16 --- a/mshadow +++ b/mshadow @@ -1 +1 @@ -Subproject commit cb5c9872b542220be5b99f3aca0e1ff56e31b028 +Subproject commit 2d7780c3f2eefe4453fa419862d1b2089bedb8d5 -- To stop receiving notification emails like this one, please contact ['"comm...@mxnet.apache.org" '].
svn commit: r23001 - /dev/incubator/mxnet/0.12.1.rc0/
Author: cjolivier01 Date: Wed Nov 8 00:44:48 2017 New Revision: 23001 Log: Added mxnet-0.12.1.rc0 Added: dev/incubator/mxnet/0.12.1.rc0/ dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz (with props) dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.asc dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.md5 dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.sha512 Added: dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz == Binary file - no diff available. Propchange: dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.asc == --- dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.asc (added) +++ dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.asc Wed Nov 8 00:44:48 2017 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQIcBAABAgAGBQJaAlMWAAoJEID9gddwPfMbAxEQALA5OFasJ/SXHIgYsEyVKhyD +7MDHsRDZy8wJG4NjT7YhotengffPdvsM9demfGWKsfbaAHI7tbIg5jfdy/JD0oqI +v5+/28IBgh06fFdOfDazx82JWMfw61M2mSjSj+AWXRhc0ELj2Z951diNEXIZRluA +3mqqGswBervK6RP0M5/s3aYggvWrWTcYFyhnfntyXl+U3xA7hiX0GVW+9U6Uosot +1/mjSELLIe/xrdw7FSypbOWIFGS6TcWBKmD0D/JsQ+e9rCeF7FIT/8mDEtvghD28 +cwDCthfuAT0QSAZag+saxxZ7YwlAkGqnAHml1X/zPseyOAk/dT5IgN7r/Fw80j7T +jRGfmV1i+N/LujB8hgdcemCcK/5NGcAmtIQ3xReloFhVV/rBXyJcBJ15WwHbf1IN +8R8DAGNs2JnvwTfbL+gHCOfDxe73M5xm1R4j+6OmnVgM7vdpgV+N0Qkfv6lfCF6g +SIpjwLO+OKcikDlHc3JNjWA1jcTERgrbN8LIUIOVX3l/sjjhheFjGs0a7prMs8J3 +omSXJpRw0YpXeMW0dimy3WzF/XdueoQUtswsvLTgKhmMAAQWpeQJzi6xjVXDOgqM +OtycvXPsjKUW5Wol8ynjWPXW916oqnMSx+S0VBoNO1MY5sW8kO/t+MvdxMoyak64 +hniL9dmukD1oS6RezbDh +=0AtJ +-END PGP SIGNATURE- Added: dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.md5 == --- dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.md5 (added) +++ dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.md5 Wed Nov 8 00:44:48 2017 @@ -0,0 +1 @@ +172758e83e71c13a40efc1493d4d3132 apache-mxnet-src-0.12.1.rc0-incubating.tar.gz Added: dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.sha512 == --- dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.sha512 (added) +++ dev/incubator/mxnet/0.12.1.rc0/apache-mxnet-src-0.12.1.rc0-incubating.tar.gz.sha512 Wed Nov 8 00:44:48 2017 @@ -0,0 +1 @@ +a0689e8c305d399ae5c28affe40ff806070f3749fa4a574ec4c06fecdc60b4f1335fcefe28ba72c804cca941bf86076ecdcaa01a5a79fa17d1085bc18f5d6e7f apache-mxnet-src-0.12.1.rc0-incubating.tar.gz
[GitHub] larroy commented on issue #8578: Add development instructions and update installation instructions
larroy commented on issue #8578: Add development instructions and update installation instructions URL: https://github.com/apache/incubator-mxnet/pull/8578#issuecomment-34266 I think is good to have an initial reference on how to build & test in the root folder, the additional documentation is more detailed. But a newcomer just checks out the project and doesn't know how to get started. I linked to the main documentation in the file in the root folder. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] larroy commented on a change in pull request #8578: Add development instructions and update installation instructions
larroy commented on a change in pull request #8578: Add development instructions and update installation instructions URL: https://github.com/apache/incubator-mxnet/pull/8578#discussion_r149545383 ## File path: docs/install/index.md ## @@ -65,7 +65,7 @@ Indicate your preferred configuration. Then, follow the customized commands to i -The following installation instructions have been tested on Ubuntu 14.04 and 16.04. Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong closed pull request #8264: Operators for mean(csr, axis=0) and mean(csr, axis=1)
piiswrong closed pull request #8264: Operators for mean(csr, axis=0) and
mean(csr, axis=1)
URL: https://github.com/apache/incubator-mxnet/pull/8264
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/operator/tensor/broadcast_reduce_op.h
b/src/operator/tensor/broadcast_reduce_op.h
index 75f96f9447..8e8b0a1fbb 100644
--- a/src/operator/tensor/broadcast_reduce_op.h
+++ b/src/operator/tensor/broadcast_reduce_op.h
@@ -566,14 +566,15 @@ struct SumCsrKernel {
}
};
-template
+/*! \brief If normalize is true, the mean should be computed instead of sum */
+template
void SumCsrImpl(const nnvm::NodeAttrs& attrs, mshadow::Stream* s, const
OpContext& ctx,
const NDArray& input, const OpReqType req, NDArray* output) {
if (req == kNullOp) return;
const ReduceAxesParam& param = nnvm::get(attrs.parsed);
- CHECK_EQ(param.axis.ndim(), 1U) << "sum(csr) only supports axis 0 or 1";
+ CHECK_EQ(param.axis.ndim(), 1U) << "sum(csr)/mean(csr) only supports axis 0
or 1";
CHECK(param.axis[0] == 0 || param.axis[0] == 1)
- << "sum(csr) only support axis 0 or 1";
+ << "sum(csr)/mean(csr) only support axis 0 or 1";
CHECK(!param.keepdims) << "keepdims not supported for sparse";
CHECK(!param.exclude) << "exclude not supported for sparse";
int64_t out_data_size = 0;
@@ -586,6 +587,7 @@ void SumCsrImpl(const nnvm::NodeAttrs& attrs,
mshadow::Stream* s, const OpC
CHECK_EQ(output->storage_type(), kDefaultStorage);
using namespace mshadow;
+ using namespace mshadow::expr;
using namespace mxnet_op;
using namespace csr;
using nnvm::dim_t;
@@ -630,19 +632,34 @@ void SumCsrImpl(const nnvm::NodeAttrs& attrs,
mshadow::Stream* s, const OpC
s, num_threads, output->data().dptr(), in_indptr,
in_idx,
in_data, sum.dptr_, residual.dptr_, num_rows, num_cols,
seg_len);
+if (normalize) {
+ mxnet_op::Kernel<
+ mxnet_op::op_with_req,
+ xpu>::Launch(s, out_data_size, output->data().dptr(),
+ output->data().dptr(), DType(num_rows));
+}
});
});
});
});
} else if (1 == param.axis[0]) {
MSHADOW_IDX_TYPE_SWITCH(input.aux_type(kIndPtr), RType, {
- MSHADOW_TYPE_SWITCH(input.dtype(), DType, {
-MXNET_ASSIGN_REQ_SWITCH(req, req_type, {
- const RType* in_indptr = input.aux_data(kIndPtr).dptr();
- const DType* in_data = input.data().dptr();
- Kernel, xpu>::Launch(
- s, out_data_size, output->data().dptr(), in_indptr,
- in_data);
+ MSHADOW_IDX_TYPE_SWITCH(input.aux_type(kIdx), IType, {
+MSHADOW_TYPE_SWITCH(input.dtype(), DType, {
+ MXNET_ASSIGN_REQ_SWITCH(req, req_type, {
+const RType* in_indptr = input.aux_data(kIndPtr).dptr();
+const DType* in_data = input.data().dptr();
+const IType num_cols = input.shape()[1];
+Kernel, xpu>::Launch(
+s, out_data_size, output->data().dptr(), in_indptr,
+in_data);
+if (normalize) {
+ mxnet_op::Kernel<
+ mxnet_op::op_with_req,
+ xpu>::Launch(s, out_data_size, output->data().dptr(),
+ output->data().dptr(), DType(num_cols));
+}
+ });
});
});
});
@@ -661,9 +678,9 @@ void SumOpForwardEx(const nnvm::NodeAttrs& attrs, const
OpContext& ctx,
const NDArrayStorageType istype = inputs[0].storage_type();
if (istype == kCSRStorage) {
CHECK_EQ(inputs[0].shape().ndim(), 2U)
-<< "sum(csr) op only supports 2D ndarray as input";
+<< "sum(csr)/mean(csr) op only supports 2D ndarray as input";
NDArray output = outputs[0];
-SumCsrImpl(attrs, s, ctx, inputs[0], req[0], &output);
+SumCsrImpl(attrs, s, ctx, inputs[0], req[0], &output);
} else {
LOG(FATAL) << "Not implemented: "
<< operator_string(attrs, ctx, inputs, req, outputs);
diff --git a/src/operator/tensor/broadcast_reduce_op_value.cc
b/src/operator/tensor/broadcast_reduce_op_value.cc
index c3644042aa..0d376c31e7 100644
--- a/src/operator/tensor/broadcast_reduce_op_value.cc
+++ b/src/operator/tensor/broadcast_reduce_op_value.cc
@@ -96,8 +96,11 @@ MXNET_OPERATOR_REGISTER_REDUCE_BACKWARD(_backward_sum)
.set_attr("FCompute", ReduceAxesBackwardUseNone);
MXNET_OPERATOR_REGISTER_REDUCE(mean)
+MXNET_ADD_SPARSE_OP_ALIAS(mean)
.describe(get_reduce_axes_description("mean", __LINE__))
.set_attr("FCompute", ReduceAxesCompute)
+.set_attr("FComputeEx", SumOpForwardEx)
+.set_attr("FInferStorageType", SumOpForwardInferStorageType)
.set_attr("FResourceReques
[incubator-mxnet] branch master updated: Operators for mean(csr, axis=0) and mean(csr, axis=1) (#8264)
This is an automated email from the ASF dual-hosted git repository.
jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c2358f9 Operators for mean(csr, axis=0) and mean(csr, axis=1) (#8264)
c2358f9 is described below
commit c2358f9b87ce51b0a39c4623f2ee3eafd3a74861
Author: Anirudh Subramanian
AuthorDate: Tue Nov 7 16:08:27 2017 -0800
Operators for mean(csr, axis=0) and mean(csr, axis=1) (#8264)
* Add Infer storage for sparse slice operator
* Remove unused files
* Indentation fix and add gpu test for fallback
* Change sum builtin to py_sum
* Add sum_axis(csr,axis=0)=dense and sum(csr,axis=1)=dense operator
* Documentation changes for sparse
* Add fallback unittest for keepdims and exclude
* PR review based changes
:
* Fix CHECK_NE
* Change in_stype to int
* Using const int instead of int
* Initialize mid with the start
* Add mean(csr, axis=0) and mean(csr, axis=1)
* Removing whitespace
* Add brief comment for normalize
* Add a comma
* Lint changes
---
src/operator/tensor/broadcast_reduce_op.h| 41 +---
src/operator/tensor/broadcast_reduce_op_value.cc | 3 ++
tests/python/unittest/test_sparse_operator.py| 20 ++--
3 files changed, 43 insertions(+), 21 deletions(-)
diff --git a/src/operator/tensor/broadcast_reduce_op.h
b/src/operator/tensor/broadcast_reduce_op.h
index 75f96f9..8e8b0a1 100644
--- a/src/operator/tensor/broadcast_reduce_op.h
+++ b/src/operator/tensor/broadcast_reduce_op.h
@@ -566,14 +566,15 @@ struct SumCsrKernel {
}
};
-template
+/*! \brief If normalize is true, the mean should be computed instead of sum */
+template
void SumCsrImpl(const nnvm::NodeAttrs& attrs, mshadow::Stream* s, const
OpContext& ctx,
const NDArray& input, const OpReqType req, NDArray* output) {
if (req == kNullOp) return;
const ReduceAxesParam& param = nnvm::get(attrs.parsed);
- CHECK_EQ(param.axis.ndim(), 1U) << "sum(csr) only supports axis 0 or 1";
+ CHECK_EQ(param.axis.ndim(), 1U) << "sum(csr)/mean(csr) only supports axis 0
or 1";
CHECK(param.axis[0] == 0 || param.axis[0] == 1)
- << "sum(csr) only support axis 0 or 1";
+ << "sum(csr)/mean(csr) only support axis 0 or 1";
CHECK(!param.keepdims) << "keepdims not supported for sparse";
CHECK(!param.exclude) << "exclude not supported for sparse";
int64_t out_data_size = 0;
@@ -586,6 +587,7 @@ void SumCsrImpl(const nnvm::NodeAttrs& attrs,
mshadow::Stream* s, const OpC
CHECK_EQ(output->storage_type(), kDefaultStorage);
using namespace mshadow;
+ using namespace mshadow::expr;
using namespace mxnet_op;
using namespace csr;
using nnvm::dim_t;
@@ -630,19 +632,34 @@ void SumCsrImpl(const nnvm::NodeAttrs& attrs,
mshadow::Stream* s, const OpC
s, num_threads, output->data().dptr(), in_indptr,
in_idx,
in_data, sum.dptr_, residual.dptr_, num_rows, num_cols,
seg_len);
+if (normalize) {
+ mxnet_op::Kernel<
+ mxnet_op::op_with_req,
+ xpu>::Launch(s, out_data_size, output->data().dptr(),
+ output->data().dptr(), DType(num_rows));
+}
});
});
});
});
} else if (1 == param.axis[0]) {
MSHADOW_IDX_TYPE_SWITCH(input.aux_type(kIndPtr), RType, {
- MSHADOW_TYPE_SWITCH(input.dtype(), DType, {
-MXNET_ASSIGN_REQ_SWITCH(req, req_type, {
- const RType* in_indptr = input.aux_data(kIndPtr).dptr();
- const DType* in_data = input.data().dptr();
- Kernel, xpu>::Launch(
- s, out_data_size, output->data().dptr(), in_indptr,
- in_data);
+ MSHADOW_IDX_TYPE_SWITCH(input.aux_type(kIdx), IType, {
+MSHADOW_TYPE_SWITCH(input.dtype(), DType, {
+ MXNET_ASSIGN_REQ_SWITCH(req, req_type, {
+const RType* in_indptr = input.aux_data(kIndPtr).dptr();
+const DType* in_data = input.data().dptr();
+const IType num_cols = input.shape()[1];
+Kernel, xpu>::Launch(
+s, out_data_size, output->data().dptr(), in_indptr,
+in_data);
+if (normalize) {
+ mxnet_op::Kernel<
+ mxnet_op::op_with_req,
+ xpu>::Launch(s, out_data_size, output->data().dptr(),
+ output->data().dptr(), DType(num_cols));
+}
+ });
});
});
});
@@ -661,9 +678,9 @@ void SumOpForwardEx(const nnvm::NodeAttrs& attrs, const
OpContext& ctx,
const NDArrayStorageType istype = inputs[0].storage_type();
if (istype == kCSRStorage) {
CHECK_EQ(inputs[0].shape().ndim(), 2U)
-
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149538454 ## File path: cmake/Blas.cmake ## @@ -0,0 +1,43 @@ +set(mshadow_LINKER_LIBS "") Review comment: i will remove it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on issue #8579: Automatic OMP operator tuning based upon kernel operation workload
cjolivier01 commented on issue #8579: Automatic OMP operator tuning based upon kernel operation workload URL: https://github.com/apache/incubator-mxnet/pull/8579#issuecomment-342663369 Closed during changes in order to not trigger CI (CI is having problems) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149538371
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
+Set(FIRST_CUDA FALSE)
+endif()
+include(cmake/Utils.cmake)
+mxnet_option(USE_CUDA "Build with CUDA support" ON)
+mxnet_option(USE_OLDCMAKECUDA "Build with old cmake cuda" OFF)
Review comment:
This just in case backup option is used to manually switch to the old mode.
Normally, it's automatically chosen
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149539798
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
+Set(FIRST_CUDA FALSE)
+endif()
+include(cmake/Utils.cmake)
+mxnet_option(USE_CUDA "Build with CUDA support" ON)
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149539745
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149539611 ## File path: cmake/FirstCuda.cmake ## @@ -0,0 +1,233 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations Review comment: this file look like `mshadow\cmake\Cuda.cmake` but it remove some content of conflict. besides, I made a mistake when I created `mshadow\cmake\Cuda.cmake`. It shouldn't have been put `mshadow\cmake\`.should be placed on `root\cmake` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149538454 ## File path: cmake/Blas.cmake ## @@ -0,0 +1,43 @@ +set(mshadow_LINKER_LIBS "") Review comment: i will remove it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149538371
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
+Set(FIRST_CUDA FALSE)
+endif()
+include(cmake/Utils.cmake)
+mxnet_option(USE_CUDA "Build with CUDA support" ON)
+mxnet_option(USE_OLDCMAKECUDA "Build with old cmake cuda" OFF)
Review comment:
This just in case backup option is used to manually switch to the old mode.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yajiedesign commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149537797
##
File path: CMakeLists.txt
##
@@ -128,14 +152,20 @@ endif()
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
-if(EXISTS ${PROJECT_SOURCE_DIR}/mshadow/cmake)
- include(mshadow/cmake/mshadow.cmake)
+if(FIRST_CUDA)
Review comment:
the ?first CUDA? mean "cmake first class language cuda".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yajiedesign commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#issuecomment-342659887 @cjolivier01 if you want use cuda9.0,the new cmake is must. and the new cmake can parallel compiling with nvcc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yajiedesign commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support
yajiedesign commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#issuecomment-342659887 @cjolivier01 if you want use cuda9.0,the new cmake is must. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8555: Update PR & Issue Template
eric-haibin-lin commented on a change in pull request #8555: Update PR & Issue Template URL: https://github.com/apache/incubator-mxnet/pull/8555#discussion_r149535629 ## File path: .github/PULL_REQUEST_TEMPLATE.md ## @@ -6,7 +6,7 @@ - [ ] Passed code style checking (`make lint`) - [ ] Changes are complete (i.e. I finished coding on this PR) - [ ] All changes have test coverage -- [ ] For user-facing API changes, API doc string has been updated. +- [ ] For user-facing API changes, API doc string has been updated. For new C++ functions in header files, their functionalities and arguments are well-documented. - [ ] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### Review comment: Cool fixed that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhreshold commented on issue #8519: More details to the windows build process
zhreshold commented on issue #8519: More details to the windows build process URL: https://github.com/apache/incubator-mxnet/pull/8519#issuecomment-342650722 Please rebase to fix CI This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin opened a new issue #8583: Improve custom cross-entropy operator example
eric-haibin-lin opened a new issue #8583: Improve custom cross-entropy operator example URL: https://github.com/apache/incubator-mxnet/issues/8583 In the recommender example, the custom cross-entropy operator can be implemented using NDArray instead of numpy, which gives much better performance. Changing the implementation to use NDArray will help other users who is learning MXNet / trying to use MXNet with custom ops. https://github.com/apache/incubator-mxnet/blob/master/example/recommenders/crossentropy.py This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on a change in pull request #8558: slice operator supporting arbitrary values of step
reminisce commented on a change in pull request #8558: slice operator
supporting arbitrary values of step
URL: https://github.com/apache/incubator-mxnet/pull/8558#discussion_r149528683
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -414,16 +420,6 @@ inline TShape GetSliceShape(const SliceParam& param,
const TShape& dshape) {
return oshape;
}
-inline bool SliceShape(const nnvm::NodeAttrs& attrs,
- std::vector *in_attrs,
- std::vector *out_attrs) {
- const TShape& dshape = (*in_attrs)[0];
- if (dshape.ndim() == 0) return false;
- const SliceParam& param = nnvm::get(attrs.parsed);
- SHAPE_ASSIGN_CHECK(*out_attrs, 0, GetSliceShape(param, dshape));
- return true;
-}
-
inline bool SliceForwardInferStorageType(const nnvm::NodeAttrs& attrs,
Review comment:
Added fallback check now. Please review again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zhreshold opened a new pull request #8582: Yolo2 pr
zhreshold opened a new pull request #8582: Yolo2 pr URL: https://github.com/apache/incubator-mxnet/pull/8582 ## Description ## Operator used by yolo2 symbol implementation. ## Checklist ## ### Essentials ### - [x] Passed code style checking (`make lint`) - [x] Changes are complete (i.e. I finished coding on this PR) - [ ] All changes have test coverage - [x] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ## Comments ## A standalone operator to support yolo v2 training using symbol and modules. Merging separately other than the full example in order to catch up with gluon tutorials. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ptrendx commented on issue #8294: NCCL integration
ptrendx commented on issue #8294: NCCL integration URL: https://github.com/apache/incubator-mxnet/pull/8294#issuecomment-342648788 Not really, no. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on issue #8294: NCCL integration
eric-haibin-lin commented on issue #8294: NCCL integration URL: https://github.com/apache/incubator-mxnet/pull/8294#issuecomment-342647982 Thanks for addressing all these review comments. Is anyone helping you to setup the CI test with NCCL? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ykim362 commented on issue #7931: MKL-DNN integration: request for reviews
ykim362 commented on issue #7931: MKL-DNN integration: request for reviews URL: https://github.com/apache/incubator-mxnet/pull/7931#issuecomment-342644868 @sbodenstein From my understanding, this PR is not going to be directly merged. It's going to be merged with another revision with sparse tensor (mkl storage). @piiswrong Is this correct? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8579: Automatic OMP operator tuning based upon kernel operation workload
cjolivier01 commented on a change in pull request #8579: Automatic OMP operator
tuning based upon kernel operation workload
URL: https://github.com/apache/incubator-mxnet/pull/8579#discussion_r149518876
##
File path: src/operator/mxnet_op.h
##
@@ -288,55 +291,84 @@ struct op_with_req {
}
};
-/*!
- * \brief Set to immediate scalar value kernel
- * \tparam val Scalar immediate
- */
-template
-struct set_to_int {
- // mxnet_op version (when used directly with Kernel<>::Launch()) */
- template
- MSHADOW_XINLINE static void Map(int i, DType* out) {
-out[i] = DType(val);
- }
- // mshadow_op version (when used with op_with_req<>)
- MSHADOW_XINLINE static int Map() {
-return val;
- }
-};
-/*! \brief Special-case kernel shortcut for setting to zero */
-using set_zero = set_to_int<0>;
+/*! \brief Kernel operator wrapper used for tuning data */
+template
+struct tuned_op : public Operation {
+ static size_t workload_; // nanos per operation * Tuner's
WORKLOAD_COUNT
+ // the decision implementation
+ // TODO(cjolivier01): For more complex kernels, add a shape parameter
version (diff LaunchEx)
+ static int UseOMP(size_t N, size_t thread_count);
+};
template
struct Kernel;
-
template
struct Kernel {
+ /*! \brief Launch CPU kernel */
template
- inline static void Launch(mshadow::Stream *s, const int N, Args...
args) {
+ inline static void Launch(mshadow::Stream *, const int N, Args... args)
{
#ifdef _OPENMP
-const int omp_cores = Engine::Get()->num_omp_threads_per_worker();
-if (omp_cores <= 1) {
+const int omp_threads =
engine::OpenMP::Get()->GetRecommendedOMPThreadCount();
+if (omp_threads < 2) {
// Zero means not to use OMP, but don't interfere with external OMP
behavior
for (int i = 0; i < N; ++i) {
OP::Map(i, args...);
}
} else {
- #pragma omp parallel for num_threads(omp_cores)
+ #pragma omp parallel for num_threads(omp_threads)
for (int i = 0; i < N; ++i) {
OP::Map(i, args...);
}
}
#else
for (int i = 0; i < N; ++i) {
+ OP::Map(i, args...);
+}
+#endif
+ }
+
+ /*! \brief Launch CPU kernel which has OMP tuning data available.
+ * When using this for a new kernel op, add declaration and tuning objects to
+ * operator_tune.cc
+ */
+ template
+ static void LaunchEx(mshadow::Stream *, const int N, DType *dest,
Args... args) {
+#ifdef _OPENMP
+const int omp_threads =
engine::OpenMP::Get()->GetRecommendedOMPThreadCount();
+if (omp_threads < 2 || !tuned_op::UseOMP(N,
omp_threads)) {
+ // Zero means not to use OMP, but don't interfere with external OMP
behavior
+ for (int i = 0; i < N; ++i) {
+OP::Map(i, dest, args...);
+ }
+} else {
+ #pragma omp parallel for num_threads(omp_threads)
+ for (int i = 0; i < N; ++i) {
+OP::Map(i, dest, args...);
+ }
+}
+#else
+for (int i = 0; i < N; ++i) {
OP::Map(i, args...);
Review comment:
dest not added here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149509544
##
File path: src/ndarray/ndarray.cc
##
@@ -1214,6 +1214,41 @@ void NDArray::SyncCopyToCPU(void *data, size_t size)
const {
}
}
+void NDArray::CheckFormat(const bool full_check) const {
+ int32_t err = kNormalErr;
+ TBlob err_cpu(&err, mshadow::Shape1(1), cpu::kDevMask, 0);
+ if (this->ctx().dev_mask() == cpu::kDevMask) {
+Engine::Get()->PushSync([&](RunContext rctx) {
+common::CheckFormatWrapper(rctx, *this, err_cpu, full_check);
+ }, this->ctx(), {this->var()}, {},
+ FnProperty::kNormal, 0, PROFILER_MESSAGE("CheckFormat"));
+this->WaitToWrite();
+ } else {
+#if MXNET_USE_CUDA
+Engine::Get()->PushSync([&](RunContext rctx) {
+common::CheckFormatWrapper(rctx, *this, err_cpu, full_check);
+rctx.get_stream()->Wait();
+ }, this->ctx(), {this->var()}, {},
+ FnProperty::kNormal, 0, PROFILER_MESSAGE("CheckFormat"));
+this->WaitToWrite();
Review comment:
We should move `this->WaitToWrite();` out side of the if else switch and use
it only once
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149509374
##
File path: src/ndarray/ndarray.cc
##
@@ -1214,6 +1214,41 @@ void NDArray::SyncCopyToCPU(void *data, size_t size)
const {
}
}
+void NDArray::CheckFormat(const bool full_check) const {
+ int32_t err = kNormalErr;
+ TBlob err_cpu(&err, mshadow::Shape1(1), cpu::kDevMask, 0);
+ if (this->ctx().dev_mask() == cpu::kDevMask) {
+Engine::Get()->PushSync([&](RunContext rctx) {
+common::CheckFormatWrapper(rctx, *this, err_cpu, full_check);
+ }, this->ctx(), {this->var()}, {},
+ FnProperty::kNormal, 0, PROFILER_MESSAGE("CheckFormat"));
+this->WaitToWrite();
+ } else {
+#if MXNET_USE_CUDA
+Engine::Get()->PushSync([&](RunContext rctx) {
+common::CheckFormatWrapper(rctx, *this, err_cpu, full_check);
+rctx.get_stream()->Wait();
+ }, this->ctx(), {this->var()}, {},
+ FnProperty::kNormal, 0, PROFILER_MESSAGE("CheckFormat"));
+this->WaitToWrite();
+#else
+LOG(FATAL) << "GPU is not enabled";
+#endif
+ }
+ CHECK_NE(err, kCSRShapeErr) << "Shape mismatch of this CSRNDArray";
+ CHECK_NE(err, kCSRIndPtrErr)
+ << "IndPtr of CSRNDArray should be non-negative, in non-decreasing
order, "
+ << "start with 0, and end with value equal with size of indices.";
+ CHECK_NE(err, kCSRIdxErr)
+ << "Indices of CSRNDArray should be non-negative, in ascending
order per row "
+ << " and less than the number of columns.";
+ CHECK_NE(err, kRSPShapeErr) << "Shape mismatch of this RSPNDArray";
+ CHECK_NE(err, kRSPIdxErr)
+ << "Indices of RSPNDArray should be non-negative, "
Review comment:
We should avoid the usage of `RSPNDArray` in the log since it's displayed to
the users. We didn't mention the term `RSPNDArray` in any other user facing
documentation. This will also be used for all language bindings. Let's replace
it with `row_sparse NDArray`/`csr NDArray` to avoid any python specific class
names.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149508628 ## File path: tests/python/unittest/test_sparse_ndarray.py ## @@ -735,6 +735,58 @@ def test_sparse_nd_exception(): assert_exception(mx.nd.sparse.zeros, ValueError, "invalid_stype", (2,2)) +def test_sparse_nd_check_format(): +""" test check_format for sparse ndarray """ +def check_csr_format_exception(data_list, indices_list, indptr_list, shape): Review comment: We can use assertRaises to make the code cleaner https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/common.py#L27 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149510743 ## File path: tests/python/unittest/test_sparse_ndarray.py ## @@ -735,6 +735,58 @@ def test_sparse_nd_exception(): assert_exception(mx.nd.sparse.zeros, ValueError, "invalid_stype", (2,2)) +def test_sparse_nd_check_format(): +""" test check_format for sparse ndarray """ +def check_csr_format_exception(data_list, indices_list, indptr_list, shape): +try: +a = mx.nd.sparse.csr_matrix((data_list, indices_list, indptr_list), shape=shape) +a.check_format() +assert(False) +except mx.base.MXNetError: +pass +def check_rsp_format_exception(data_list, indices_list, shape): +try: +a = mx.nd.sparse.row_sparse_array((data_list, indices_list), shape=shape) +a.check_format() +assert(False) +except mx.base.MXNetError: +pass + +shape = rand_shape_2d() +stypes = ["csr", "row_sparse"] +for stype in stypes: +arr, _ = rand_sparse_ndarray(shape, stype) +arr.check_format() +# CSR format index pointer array should be less than the number of rows Review comment: Do we also have test cases for zeros/empty csr/rsp ndarray? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149510310
##
File path: src/common/utils.h
##
@@ -43,9 +43,170 @@
#include
#include
+#include "../operator/mxnet_op.h"
+
namespace mxnet {
namespace common {
+
+/*!
+ * \brief IndPtr should be non-negative, in non-decreasing order, start with 0
+ * and end with value equal with size of indices.
+ */
+struct csr_indptr_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* indptr,
+ const nnvm::dim_t end, const nnvm::dim_t
idx_size) {
+if (indptr[i+1] < 0 || indptr[i+1] < indptr[i] ||
+(i == 0 && indptr[i] != 0) ||
+(i == end - 1 && indptr[end] != idx_size))
+ *out = kCSRIndPtrErr;
+ }
+};
+
+/*!
+ * \brief Indices should be non-negative, less than the number of columns
+ * and in ascending order per row.
+ */
+struct csr_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const RType* indptr, const nnvm::dim_t
ncols) {
+for (RType j = indptr[i]; j < indptr[i+1]; j++) {
+ if (idx[j] >= ncols || idx[j] < 0 ||
+ (j < indptr[i+1] - 1 && idx[j] >= idx[j+1])) {
+*out = kCSRIdxErr;
+break;
+ }
+}
+ }
+};
+
+/*!
+ * \brief Indices of RSPNDArray should be non-negative,
+ * less than the size of first dimension and in ascending order
+ */
+struct rsp_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const nnvm::dim_t end, const nnvm::dim_t
nrows) {
+if ((i < end && idx[i+1] <= idx[i])
+|| idx[i] < 0 || idx[i] >= nrows)
+ *out = kRSPIdxErr;
+ }
+};
+
+template
+void CheckFormatWrapper(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check);
+
+/*!
+ * \brief Check the validity of CSRNDArray.
+ * \param rctx Execution context.
+ * \param input Input NDArray of CSRStorage.
+ * \param err_cpu Error number on cpu.
+ * \param full_check If true, rigorous check, O(N) operations,
+ * otherwise basic check, O(1) operations.
+ */
+template
+void CheckFormatCSRImpl(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check) {
+ using namespace op::mxnet_op;
+ CHECK_EQ(input.storage_type(), kCSRStorage)
+ << "CheckFormatCSRImpl is for CSRNDArray";
+ const TShape shape = input.shape();
+ const TShape idx_shape = input.aux_shape(csr::kIdx);
+ const TShape indptr_shape = input.aux_shape(csr::kIndPtr);
+ const TShape storage_shape = input.storage_shape();
+ if ((shape.ndim() != 2) ||
+ (idx_shape.ndim() != 1 || indptr_shape.ndim() != 1 ||
storage_shape.ndim() != 1) ||
+ (indptr_shape[0] != shape[0] + 1) ||
+ (idx_shape[0] != storage_shape[0])) {
+ MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ auto err = err_cpu.dptr();
+ *err = kCSRShapeErr;
+ });
+ return;
+ }
+ if (full_check) {
+MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIndPtr), RType, {
+MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIdx), IType, {
+ mshadow::Stream *s = rctx.get_stream();
+ NDArray ret_xpu = NDArray(mshadow::Shape1(1),
+rctx.get_ctx(), false, err_cpu.type_flag_);
+ TBlob val_xpu = ret_xpu.data();
+ Kernel, xpu>::Launch(s, val_xpu.Size(),
val_xpu.dptr());
+ Kernel::Launch(s, indptr_shape[0] - 1,
val_xpu.dptr(),
+input.aux_data(csr::kIndPtr).dptr(),
+indptr_shape[0] - 1, idx_shape[0]);
+ Kernel::Launch(s, indptr_shape[0] - 1,
val_xpu.dptr(),
+input.aux_data(csr::kIdx).dptr(),
+input.aux_data(csr::kIndPtr).dptr(), shape[1]);
+ mshadow::Copy(err_cpu.get(),
+val_xpu.get(s), s);
+});
+ });
+});
+ }
+}
+
+/*!
+ * \brief Check the validity of RowSparseNDArray.
+ * \param rctx Execution context.
+ * \param input Input NDArray of RowSparseStorage.
+ * \param err_cpu Error number on cpu.
+ * \param full_check If true, rigorous check, O(N) operations,
+ * otherwise basic check, O(1) operations.
+ */
+template
+void CheckFormatRSPImpl(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check) {
+ using namespace op::mxnet_op;
+ CHECK_EQ(input.storage_type(), kRowSparseStorage)
+ << "CheckFormatRSPImpl is for RSPNDArray";
+ const TShape idx_shape = input.aux_shape(rowsparse::kIdx);
+ if (idx_shape[0] != input.storage_shape()[0]) {
+MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ auto err = err_cpu.dptr();
Re
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149509978
##
File path: src/common/utils.h
##
@@ -43,9 +43,170 @@
#include
#include
+#include "../operator/mxnet_op.h"
+
namespace mxnet {
namespace common {
+
+/*!
+ * \brief IndPtr should be non-negative, in non-decreasing order, start with 0
+ * and end with value equal with size of indices.
+ */
+struct csr_indptr_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* indptr,
+ const nnvm::dim_t end, const nnvm::dim_t
idx_size) {
+if (indptr[i+1] < 0 || indptr[i+1] < indptr[i] ||
+(i == 0 && indptr[i] != 0) ||
+(i == end - 1 && indptr[end] != idx_size))
+ *out = kCSRIndPtrErr;
+ }
+};
+
+/*!
+ * \brief Indices should be non-negative, less than the number of columns
+ * and in ascending order per row.
+ */
+struct csr_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const RType* indptr, const nnvm::dim_t
ncols) {
+for (RType j = indptr[i]; j < indptr[i+1]; j++) {
+ if (idx[j] >= ncols || idx[j] < 0 ||
+ (j < indptr[i+1] - 1 && idx[j] >= idx[j+1])) {
+*out = kCSRIdxErr;
+break;
+ }
+}
+ }
+};
+
+/*!
+ * \brief Indices of RSPNDArray should be non-negative,
+ * less than the size of first dimension and in ascending order
+ */
+struct rsp_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const nnvm::dim_t end, const nnvm::dim_t
nrows) {
+if ((i < end && idx[i+1] <= idx[i])
+|| idx[i] < 0 || idx[i] >= nrows)
+ *out = kRSPIdxErr;
+ }
+};
+
+template
+void CheckFormatWrapper(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check);
+
+/*!
+ * \brief Check the validity of CSRNDArray.
+ * \param rctx Execution context.
+ * \param input Input NDArray of CSRStorage.
+ * \param err_cpu Error number on cpu.
+ * \param full_check If true, rigorous check, O(N) operations,
+ * otherwise basic check, O(1) operations.
+ */
+template
+void CheckFormatCSRImpl(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check) {
+ using namespace op::mxnet_op;
+ CHECK_EQ(input.storage_type(), kCSRStorage)
+ << "CheckFormatCSRImpl is for CSRNDArray";
+ const TShape shape = input.shape();
+ const TShape idx_shape = input.aux_shape(csr::kIdx);
+ const TShape indptr_shape = input.aux_shape(csr::kIndPtr);
+ const TShape storage_shape = input.storage_shape();
+ if ((shape.ndim() != 2) ||
+ (idx_shape.ndim() != 1 || indptr_shape.ndim() != 1 ||
storage_shape.ndim() != 1) ||
+ (indptr_shape[0] != shape[0] + 1) ||
+ (idx_shape[0] != storage_shape[0])) {
+ MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ auto err = err_cpu.dptr();
+ *err = kCSRShapeErr;
+ });
+ return;
+ }
+ if (full_check) {
+MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIndPtr), RType, {
+MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIdx), IType, {
+ mshadow::Stream *s = rctx.get_stream();
+ NDArray ret_xpu = NDArray(mshadow::Shape1(1),
+rctx.get_ctx(), false, err_cpu.type_flag_);
+ TBlob val_xpu = ret_xpu.data();
+ Kernel, xpu>::Launch(s, val_xpu.Size(),
val_xpu.dptr());
+ Kernel::Launch(s, indptr_shape[0] - 1,
val_xpu.dptr(),
+input.aux_data(csr::kIndPtr).dptr(),
+indptr_shape[0] - 1, idx_shape[0]);
+ Kernel::Launch(s, indptr_shape[0] - 1,
val_xpu.dptr(),
+input.aux_data(csr::kIdx).dptr(),
+input.aux_data(csr::kIndPtr).dptr(), shape[1]);
+ mshadow::Copy(err_cpu.get(),
+val_xpu.get(s), s);
+});
+ });
+});
+ }
+}
+
+/*!
+ * \brief Check the validity of RowSparseNDArray.
+ * \param rctx Execution context.
+ * \param input Input NDArray of RowSparseStorage.
+ * \param err_cpu Error number on cpu.
+ * \param full_check If true, rigorous check, O(N) operations,
+ * otherwise basic check, O(1) operations.
+ */
+template
+void CheckFormatRSPImpl(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check) {
+ using namespace op::mxnet_op;
+ CHECK_EQ(input.storage_type(), kRowSparseStorage)
+ << "CheckFormatRSPImpl is for RSPNDArray";
+ const TShape idx_shape = input.aux_shape(rowsparse::kIdx);
+ if (idx_shape[0] != input.storage_shape()[0]) {
+MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ auto err = err_cpu.dptr();
+
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149507608 ## File path: include/mxnet/c_api.h ## @@ -405,6 +405,12 @@ MXNET_DLL int MXNDArraySyncCopyFromNDArray(NDArrayHandle handle_dst, const int i); /*! + * \brief check whether the matrix format is valid + * \param full_check if `True`, rigorous check, O(N) operations + *Otherwise basic check, O(1) operations + */ +MXNET_DLL int MXNDArrayCheckFormat(NDArrayHandle handle, const bool full_check); Review comment: Let's rename the c_api to MXNDArraySyncCheckFormat This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149507450
##
File path: include/mxnet/ndarray.h
##
@@ -447,6 +456,12 @@ class NDArray {
*/
void SyncCopyToCPU(void *data, size_t size) const;
/*!
+ * \brief check whether the matrix format is valid
+ * \param full_check if `True`, rigorous check, O(N) operations
+ *Otherwise basic check, O(1) operations
+ */
+ void CheckFormat(const bool full_check) const;
Review comment:
I think we'd better rename this function to `SyncCheckFormat`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8259: check_format of sparse ndrray
eric-haibin-lin commented on a change in pull request #8259: check_format of
sparse ndrray
URL: https://github.com/apache/incubator-mxnet/pull/8259#discussion_r149516527
##
File path: src/common/utils.h
##
@@ -43,9 +43,170 @@
#include
#include
+#include "../operator/mxnet_op.h"
+
namespace mxnet {
namespace common {
+
+/*!
+ * \brief IndPtr should be non-negative, in non-decreasing order, start with 0
+ * and end with value equal with size of indices.
+ */
+struct csr_indptr_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* indptr,
+ const nnvm::dim_t end, const nnvm::dim_t
idx_size) {
+if (indptr[i+1] < 0 || indptr[i+1] < indptr[i] ||
+(i == 0 && indptr[i] != 0) ||
+(i == end - 1 && indptr[end] != idx_size))
+ *out = kCSRIndPtrErr;
+ }
+};
+
+/*!
+ * \brief Indices should be non-negative, less than the number of columns
+ * and in ascending order per row.
+ */
+struct csr_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const RType* indptr, const nnvm::dim_t
ncols) {
+for (RType j = indptr[i]; j < indptr[i+1]; j++) {
+ if (idx[j] >= ncols || idx[j] < 0 ||
+ (j < indptr[i+1] - 1 && idx[j] >= idx[j+1])) {
+*out = kCSRIdxErr;
+break;
+ }
+}
+ }
+};
+
+/*!
+ * \brief Indices of RSPNDArray should be non-negative,
+ * less than the size of first dimension and in ascending order
+ */
+struct rsp_idx_check {
+ template
+ MSHADOW_XINLINE static void Map(int i, DType* out, const IType* idx,
+ const nnvm::dim_t end, const nnvm::dim_t
nrows) {
+if ((i < end && idx[i+1] <= idx[i])
+|| idx[i] < 0 || idx[i] >= nrows)
+ *out = kRSPIdxErr;
+ }
+};
+
+template
+void CheckFormatWrapper(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check);
+
+/*!
+ * \brief Check the validity of CSRNDArray.
+ * \param rctx Execution context.
+ * \param input Input NDArray of CSRStorage.
+ * \param err_cpu Error number on cpu.
+ * \param full_check If true, rigorous check, O(N) operations,
+ * otherwise basic check, O(1) operations.
+ */
+template
+void CheckFormatCSRImpl(const RunContext &rctx, const NDArray &input,
+const TBlob &err_cpu, const bool full_check) {
+ using namespace op::mxnet_op;
+ CHECK_EQ(input.storage_type(), kCSRStorage)
+ << "CheckFormatCSRImpl is for CSRNDArray";
+ const TShape shape = input.shape();
+ const TShape idx_shape = input.aux_shape(csr::kIdx);
+ const TShape indptr_shape = input.aux_shape(csr::kIndPtr);
+ const TShape storage_shape = input.storage_shape();
+ if ((shape.ndim() != 2) ||
+ (idx_shape.ndim() != 1 || indptr_shape.ndim() != 1 ||
storage_shape.ndim() != 1) ||
+ (indptr_shape[0] != shape[0] + 1) ||
+ (idx_shape[0] != storage_shape[0])) {
+ MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ auto err = err_cpu.dptr();
+ *err = kCSRShapeErr;
+ });
+ return;
+ }
+ if (full_check) {
+MSHADOW_TYPE_SWITCH(err_cpu.type_flag_, DType, {
+ MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIndPtr), RType, {
+MSHADOW_IDX_TYPE_SWITCH(input.aux_type(csr::kIdx), IType, {
+ mshadow::Stream *s = rctx.get_stream();
+ NDArray ret_xpu = NDArray(mshadow::Shape1(1),
+rctx.get_ctx(), false, err_cpu.type_flag_);
+ TBlob val_xpu = ret_xpu.data();
+ Kernel, xpu>::Launch(s, val_xpu.Size(),
val_xpu.dptr());
+ Kernel::Launch(s, indptr_shape[0] - 1,
val_xpu.dptr(),
Review comment:
Indptr should always be checked, but we should avoid launching cuda kernels
for zeros csr ndarray where the indices array is empty.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on issue #7335: Exception in threads kills entire process
eric-haibin-lin commented on issue #7335: Exception in threads kills entire process URL: https://github.com/apache/incubator-mxnet/issues/7335#issuecomment-342635531 @anirudh2290 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mccollum-amzn opened a new issue #7335: Exception in threads kills entire process
mccollum-amzn opened a new issue #7335: Exception in threads kills entire process URL: https://github.com/apache/incubator-mxnet/issues/7335 For bugs or installation issues, please provide the following information. The more information you provide, the more likely people will be able to help you. ## Environment info Operating System: MacOS Compiler: Clang Package used (Python/R/Scala/Julia): Python MXNet commit hash (`git rev-parse HEAD`): 3a48185324a47d988ba6cef3bfbdcaecd30793a0 ## Error Message: I am looking at some code that adds new operators to MXNet. In a few edge cases, this code uses the CHECK macros to assert certain properties. When a CHECK fails, it throws an exception using the LOG_FATAL macro. This exception makes its way up to ExecuteOprBlock() in the ThreadedEngine class. From here, it is logged using LOG_ERROR. This causes the exception to be printed out to the console (twice actually, due to a simple MXNet bug) and then another exception is thrown out of the thread?s run handler. Following the C++ spec, this second exception causes terminate() to be called on the entire process, exiting MXNet. This has a few side-effects I?d like some feedback on. First, the caught exception is only ever logged to the console. Anyone using Jupyter will never see any errors unless they have access to the console that launched the kernel. If you are using a hosted notebook solution, where you don?t see the console, the process will exit and zero information will be provided back to the user. This is a pretty awful user-experience. Second, the environment itself exits. If you were a few days into training a model when the problem occurs, all your work will be lost. You?ll see a stack-trace, but you?ll be forced to start everything over again. Third, this means that MXNet behaves very different between the NaiveEngine and the regular threaded engine. In Na?ve mode, the exception is printed inside the interpreter and your environment is retained. In Threaded mode, the exception is only logged to the console, the interpreter exits, and you lose all your work. Are these behaviours we want to keep? It seems to me that the proper thing would be for the ThreadedEngine to catch the exception and pass it back to the main thread where it could be treated the same as exceptions in the main thread. We could use C++11 exception_ptr (http://en.cppreference.com/w/cpp/error/exception_ptr) to pass the exceptions back to the main thread. One problem is that we need to ensure related operations in other threads are also terminated. Something like: 1. Print out related operands to a log file. 2. Pass the exception to the main thread for processing and display. 3. Kill the operators depending on the operator that threw the exception. ## Minimum reproducible example Modify any operator that executes inside a thread to include something like: ``` CHECK(0) << "Exception thrown here"; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong closed pull request #8460: cpu sparse embedding op
piiswrong closed pull request #8460: cpu sparse embedding op
URL: https://github.com/apache/incubator-mxnet/pull/8460
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/example/sparse/get_data.py b/example/sparse/get_data.py
index 21db06d8e7..e96fd804b6 100644
--- a/example/sparse/get_data.py
+++ b/example/sparse/get_data.py
@@ -15,9 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-# pylint: skip-file
import os, gzip
import sys
+import mxnet as mx
+
+class DummyIter(mx.io.DataIter):
+"A dummy iterator that always return the same batch, used for speed
testing"
+def __init__(self, real_iter):
+super(DummyIter, self).__init__()
+self.real_iter = real_iter
+self.provide_data = real_iter.provide_data
+self.provide_label = real_iter.provide_label
+self.batch_size = real_iter.batch_size
+
+for batch in real_iter:
+self.the_batch = batch
+break
+
+def __iter__(self):
+return self
+
+def next(self):
+return self.the_batch
def get_libsvm_data(data_dir, data_name, url):
if not os.path.isdir(data_dir):
@@ -31,3 +50,42 @@ def get_libsvm_data(data_dir, data_name, url):
os.system("bzip2 -d %r" % data_name + ".bz2")
print("Dataset " + data_name + " is now present.")
os.chdir("..")
+
+def get_movielens_data(prefix):
+if not os.path.exists("%s.zip" % prefix):
+print("Dataset MovieLens 10M not present. Downloading now ...")
+os.system("wget http://files.grouplens.org/datasets/movielens/%s.zip";
% prefix)
+os.system("unzip %s.zip" % prefix)
+os.system("cd ml-10M100K; sh split_ratings.sh; cd -;")
+
+def get_movielens_iter(filename, batch_size, dummy_iter):
+"""Not particularly fast code to parse the text file and load into
NDArrays.
+return two data iters, one for train, the other for validation.
+"""
+print("Preparing data iterators for " + filename + " ... ")
+user = []
+item = []
+score = []
+with file(filename) as f:
+num_samples = 0
+for line in f:
+tks = line.strip().split('::')
+if len(tks) != 4:
+continue
+num_samples += 1
+user.append((tks[0]))
+item.append((tks[1]))
+score.append((tks[2]))
+if dummy_iter and num_samples > batch_size * 10:
+break
+# convert to ndarrays
+user = mx.nd.array(user, dtype='int32')
+item = mx.nd.array(item)
+score = mx.nd.array(score)
+# prepare data iters
+data_train = {'user':user, 'item':item}
+label_train = {'score':score}
+iter_train = mx.io.NDArrayIter(data=data_train,label=label_train,
+ batch_size=batch_size, shuffle=True)
+iter_train = DummyIter(iter_train) if dummy_iter else iter_train
+return mx.io.PrefetchingIter(iter_train)
diff --git a/example/sparse/matrix_fact_model.py
b/example/sparse/matrix_fact_model.py
new file mode 100644
index 00..d2d8de5dd3
--- /dev/null
+++ b/example/sparse/matrix_fact_model.py
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet as mx
+
+def matrix_fact_net(factor_size, num_hidden, max_user, max_item,
sparse_embed=True):
+# input
+user = mx.symbol.Variable('user')
+item = mx.symbol.Variable('item')
+score = mx.symbol.Variable('score')
+if sparse_embed:
+# user feature lookup
+user_weight = mx.symbol.Variable('user_weight', stype='row_sparse')
+user = mx.symbol.contrib.SparseEmbedding(data=user, weight=user_weight,
+ input_dim=max_user,
output_dim=factor_size)
+# item feature lookup
+item_weight = mx.symbol.Variable('item_weight', stype='row_sparse')
+item = mx.symbol.contrib.SparseEmbedding(data=item, weight=item_weight,
+ inpu
[incubator-mxnet] branch master updated: cpu sparse embedding op (#8460)
This is an automated email from the ASF dual-hosted git repository.
jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 4862c41 cpu sparse embedding op (#8460)
4862c41 is described below
commit 4862c4198404ad0469be7e520e5ae69e46fe862d
Author: Haibin Lin
AuthorDate: Tue Nov 7 13:52:15 2017 -0800
cpu sparse embedding op (#8460)
* cpu embedding draft
* clean up
* fix omp thread call
* add sparse embedding example
* check bound with signel thread
* add note
* add comments
* add operator note
* support rsp weight sharing for bucketing
* improve workload balance in take add grad rsp kernel
* use MSHADOW_CINLINE for cpu kernel
* review comments. add unit test for shared rsp weight
* remove indexing op-inl.h
* Trigger
* Trigger
---
example/sparse/get_data.py| 60 +++-
example/sparse/matrix_fact_model.py | 51
example/sparse/matrix_factorization.py| 111
example/sparse/readme.md | 11 +-
src/executor/graph_executor.cc| 76 +++--
src/ndarray/ndarray.cc| 8 +-
src/ndarray/ndarray_function.cu | 1 +
src/operator/tensor/cast_storage-inl.cuh | 2 +
src/operator/tensor/dot-inl.cuh | 1 +
src/operator/tensor/indexing_op.cc| 82 ++
src/operator/tensor/indexing_op.h | 388 --
src/operator/tensor/util/tensor_util-inl.cuh | 26 --
src/operator/tensor/util/tensor_util-inl.h| 82 ++
tests/python/unittest/test_module.py | 101 ---
tests/python/unittest/test_sparse_operator.py | 40 +++
15 files changed, 926 insertions(+), 114 deletions(-)
diff --git a/example/sparse/get_data.py b/example/sparse/get_data.py
index 21db06d..e96fd80 100644
--- a/example/sparse/get_data.py
+++ b/example/sparse/get_data.py
@@ -15,9 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-# pylint: skip-file
import os, gzip
import sys
+import mxnet as mx
+
+class DummyIter(mx.io.DataIter):
+"A dummy iterator that always return the same batch, used for speed
testing"
+def __init__(self, real_iter):
+super(DummyIter, self).__init__()
+self.real_iter = real_iter
+self.provide_data = real_iter.provide_data
+self.provide_label = real_iter.provide_label
+self.batch_size = real_iter.batch_size
+
+for batch in real_iter:
+self.the_batch = batch
+break
+
+def __iter__(self):
+return self
+
+def next(self):
+return self.the_batch
def get_libsvm_data(data_dir, data_name, url):
if not os.path.isdir(data_dir):
@@ -31,3 +50,42 @@ def get_libsvm_data(data_dir, data_name, url):
os.system("bzip2 -d %r" % data_name + ".bz2")
print("Dataset " + data_name + " is now present.")
os.chdir("..")
+
+def get_movielens_data(prefix):
+if not os.path.exists("%s.zip" % prefix):
+print("Dataset MovieLens 10M not present. Downloading now ...")
+os.system("wget http://files.grouplens.org/datasets/movielens/%s.zip";
% prefix)
+os.system("unzip %s.zip" % prefix)
+os.system("cd ml-10M100K; sh split_ratings.sh; cd -;")
+
+def get_movielens_iter(filename, batch_size, dummy_iter):
+"""Not particularly fast code to parse the text file and load into
NDArrays.
+return two data iters, one for train, the other for validation.
+"""
+print("Preparing data iterators for " + filename + " ... ")
+user = []
+item = []
+score = []
+with file(filename) as f:
+num_samples = 0
+for line in f:
+tks = line.strip().split('::')
+if len(tks) != 4:
+continue
+num_samples += 1
+user.append((tks[0]))
+item.append((tks[1]))
+score.append((tks[2]))
+if dummy_iter and num_samples > batch_size * 10:
+break
+# convert to ndarrays
+user = mx.nd.array(user, dtype='int32')
+item = mx.nd.array(item)
+score = mx.nd.array(score)
+# prepare data iters
+data_train = {'user':user, 'item':item}
+label_train = {'score':score}
+iter_train = mx.io.NDArrayIter(data=data_train,label=label_train,
+ batch_size=batch_size, shuffle=True)
+iter_train = DummyIter(iter_train) if dummy_iter else iter_train
+return mx.io.PrefetchingIter(iter_train)
diff --git a/example/sparse/matrix_fact_model.py
b/example/sparse/matrix_fact_model.py
new file mode 100644
index 000..d2d8de5
--- /dev/null
+++ b/example/sparse/matrix_fact_model.py
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Founda
[GitHub] SrivastavaKshitij opened a new issue #8581: More examples ?
SrivastavaKshitij opened a new issue #8581: More examples ? URL: https://github.com/apache/incubator-mxnet/issues/8581 Hi There I am trying to use mxnet and gluon for text classification using cnn. I can't find extensive examples for mxnet. All the gluon tutorials , show models which have been initialized as Net = gluon.nn.Sequential(). I would like to see examples which have been created using block or hybrid class. Can somebody redirect me to where I can find such examples This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rahul003 closed pull request #8580: fix some memory leaks, and initializes a variable
rahul003 closed pull request #8580: fix some memory leaks, and initializes a
variable
URL: https://github.com/apache/incubator-mxnet/pull/8580
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/cpp-package/example/charRNN.cpp b/cpp-package/example/charRNN.cpp
index 218d11efc9..c0239f9694 100644
--- a/cpp-package/example/charRNN.cpp
+++ b/cpp-package/example/charRNN.cpp
@@ -501,6 +501,7 @@ void train(const string file, int batch_size, int
max_epoch, int start_epoch) {
string filepath = prefix + "-" + to_string(epoch) + ".params";
SaveCheckpoint(filepath, RNN, exe);
}
+ delete exe;
}
/*The original example, rnn_cell_demo.py, uses default Xavier as initalizer,
which relies on
@@ -580,6 +581,7 @@ void trainWithBuiltInRNNOp(const string file, int
batch_size, int max_epoch, int
string filepath = prefix + "-" + to_string(epoch) + ".params";
SaveCheckpoint(filepath, RNN, exe);
}
+ delete exe;
}
void predict(wstring* ptext, int sequence_length, const string param_file,
@@ -642,6 +644,7 @@ void predict(wstring* ptext, int sequence_length, const
string param_file,
next = charIndices[n];
ptext->push_back(next);
}
+ delete exe;
}
void predictWithBuiltInRNNOp(wstring* ptext, int sequence_length, const string
param_file,
@@ -696,6 +699,7 @@ void predictWithBuiltInRNNOp(wstring* ptext, int
sequence_length, const string p
next = charIndices[n];
ptext->push_back(next);
}
+ delete exe;
}
int main(int argc, char** argv) {
diff --git a/src/engine/profiler.cc b/src/engine/profiler.cc
index 99504f61ce..20b4d2cec1 100644
--- a/src/engine/profiler.cc
+++ b/src/engine/profiler.cc
@@ -71,6 +71,10 @@ Profiler::Profiler()
}
}
+Profiler::~Profiler(){
+ delete this->profile_stat;
+}
+
Profiler* Profiler::Get() {
#if MXNET_USE_PROFILER
static Profiler inst;
@@ -112,6 +116,7 @@ OprExecStat *Profiler::AddOprStat(int dev_type, uint32_t
dev_id) {
idx = cpu_num_ + gpu_num_;
break;
default:
+ delete opr_stat;
LOG(FATAL) << "Unkown dev_type";
return NULL;
}
diff --git a/src/engine/profiler.h b/src/engine/profiler.h
index b7f8e0e1f0..334984b8b4 100644
--- a/src/engine/profiler.h
+++ b/src/engine/profiler.h
@@ -117,6 +117,7 @@ class Profiler {
protected:
/*! \brief make constructor protected. */
Profiler();
+ ~Profiler();
private:
/*! \brief generate device information following chrome profile file format
*/
diff --git a/src/engine/stream_manager.h b/src/engine/stream_manager.h
index cd6db53f14..c4633c1da7 100644
--- a/src/engine/stream_manager.h
+++ b/src/engine/stream_manager.h
@@ -90,6 +90,9 @@ RunContext StreamManager::GetRunContext(
LOG(FATAL) << MXNET_GPU_NOT_ENABLED_ERROR;
#endif // MXNET_USE_CUDA
}
+default:
+ LOG(FATAL) << "Undefined dev_mask "<::GetIORunContext(
LOG(FATAL) << MXNET_GPU_NOT_ENABLED_ERROR;
#endif // MXNET_USE_CUDA
}
+default:
+ LOG(FATAL) << "Undefined dev_mask "<(UpSamplingParam param, int dtype) {
p.stride = TShape(shape, shape + 2);
shape[0] = shape[1] = pad;
p.pad = TShape(shape, shape + 2);
+ p.cudnn_off = false;
op = new DeconvolutionOp(p);
} else {
LOG(FATAL) << "Unknown sample type";
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] rahul003 opened a new pull request #8580: fix some memory leaks, and uninitialized variable errors
rahul003 opened a new pull request #8580: fix some memory leaks, and uninitialized variable errors URL: https://github.com/apache/incubator-mxnet/pull/8580 ## Description ## 1. Fixes memory leaks 2. Sets value to uninitialized variable 3. Adds a default case to a switch statement ## Checklist ## ### Essentials ### - [ ] Passed code style checking (`make lint`) - [ ] Changes are complete (i.e. I finished coding on this PR) - [ ] All changes have test coverage - [ ] For user-facing API changes, API doc string has been updated. - [ ] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [ ] Feature1, tests, (and when applicable, API doc) - [ ] Feature2, tests, (and when applicable, API doc) ## Comments ## - If this change is a backward incompatible change, why must this change be made. - Intersting edge cases to note here This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8558: slice operator supporting arbitrary values of step
eric-haibin-lin commented on a change in pull request #8558: slice operator
supporting arbitrary values of step
URL: https://github.com/apache/incubator-mxnet/pull/8558#discussion_r149505905
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -414,16 +420,6 @@ inline TShape GetSliceShape(const SliceParam& param,
const TShape& dshape) {
return oshape;
}
-inline bool SliceShape(const nnvm::NodeAttrs& attrs,
- std::vector *in_attrs,
- std::vector *out_attrs) {
- const TShape& dshape = (*in_attrs)[0];
- if (dshape.ndim() == 0) return false;
- const SliceParam& param = nnvm::get(attrs.parsed);
- SHAPE_ASSIGN_CHECK(*out_attrs, 0, GetSliceShape(param, dshape));
- return true;
-}
-
inline bool SliceForwardInferStorageType(const nnvm::NodeAttrs& attrs,
Review comment:
Does the slice implementation for CSR support arbitrary step size? Do we
want to update the infer storage fucntion to fallback appropriately?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for csr on two dimensions, cpu implementation
eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for
csr on two dimensions, cpu implementation
URL: https://github.com/apache/incubator-mxnet/pull/8331#discussion_r149505376
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -601,18 +595,151 @@ void SliceCsrImpl(const SliceParam ¶m, const
OpContext& ctx,
});
}
+/*!
+ * \brief slice a CSRNDArray for two dimensions
+ */
+struct SliceDimTwoCsrAssign {
+ /*!
+ * \brief This function slices a CSRNDArray on axis one between begin_col
and end_col
+ * \param i loop index
+ * \param out_idx output csr ndarray column indices
+ * \param out_dataoutput csr ndarray data
+ * \param out_indptr output csr ndarray row index pointer
+ * \param in_idx input csr ndarray column indices
+ * \param in_data input csr ndarray data
+ * \param in_indptr input csr ndarray row index pointer
+ * \param begin_col begin column indice
+ * \param end_col end column indice
+ */
+ template
+ MSHADOW_XINLINE static void Map(int i,
+ IType* out_idx, DType* out_data,
+ const RType* out_indptr,
+ const IType* in_idx, const DType* in_data,
+ const RType* in_indptr,
+ const int begin_col, const int end_col) {
+RType ind = out_indptr[i];
+for (RType j = in_indptr[i]; j < in_indptr[i+1]; j++) {
+ // indices of CSRNDArray are in ascending order per row
+ if (in_idx[j] >= end_col) {
+break;
+ } else if (in_idx[j] >= begin_col) {
+out_idx[ind] = in_idx[j] - begin_col;
+out_data[ind] = in_data[j];
+ind++;
+ }
+}
+ }
+};
+
+/*
+ * Slice a CSR NDArray for two dimensions
+ * Only implemented for CPU
+ */
+template
+void SliceDimTwoCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
+ using namespace mshadow;
+ using namespace mxnet_op;
+ using namespace csr;
+ CHECK((std::is_same::value)) << "SliceDimTwoCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0], end_row = end[0];
+ nnvm::dim_t begin_col = begin[1], end_col = end[1];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
+ out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
+ // assume idx indptr share the same type
+ MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
+MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
+ MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
+RType *in_indptr = in.aux_data(kIndPtr).dptr();
+IType *in_idx = in.aux_data(kIdx).dptr();
+DType *in_data = in.data().dptr();
+// retrieve nnz (CPU implementation)
+RType *out_indptr = out.aux_data(kIndPtr).dptr();
+int nnz = 0;
+out_indptr[0] = 0;
+// loop through indptr array and corresponding indices to count for nnz
+for (nnvm::dim_t i = 0; i < indptr_len - 1; i++) {
+ out_indptr[i+1] = out_indptr[i];
+ for (RType j = in_indptr[i + begin_row];
+ j < in_indptr[i + begin_row + 1]; j++) {
+// indices of CSRNDArray are in ascending order per row
+if (in_idx[j] >= end_col) {
+ break;
+} else if (in_idx[j] >= begin_col) {
+ out_indptr[i+1]++;
+ nnz++;
+}
+ }
+}
+// returns zeros in csr format if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
Review comment:
Same comment here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for csr on two dimensions, cpu implementation
eric-haibin-lin commented on a change in pull request #8331: [sparse] slice for
csr on two dimensions, cpu implementation
URL: https://github.com/apache/incubator-mxnet/pull/8331#discussion_r149505293
##
File path: src/operator/tensor/matrix_op-inl.h
##
@@ -551,48 +550,43 @@ void SliceCsrIndPtrImpl(const int begin, const int end,
RunContext ctx,
}
/*
- * Slice a CSR NDArray
+ * Slice a CSR NDArray for first dimension
* Only implemented for CPU
*/
template
-void SliceCsrImpl(const SliceParam ¶m, const OpContext& ctx,
- const NDArray &in, OpReqType req, const NDArray &out) {
+void SliceDimOneCsrImpl(const TShape &begin, const TShape &end, const
OpContext& ctx,
+const NDArray &in, const NDArray &out) {
using namespace mshadow;
using namespace mxnet_op;
using namespace csr;
- CHECK((std::is_same::value)) << "Slice for CSR input only
implemented for CPU";
- if (req == kNullOp) return;
- CHECK_NE(req, kAddTo) << "kAddTo for Slice on CSR input is not supported";
- CHECK_NE(req, kWriteInplace) << "kWriteInplace for Slice on CSR input is not
supported";
- const TShape ishape = in.shape();
- int begin = *param.begin[0];
- if (begin < 0) begin += ishape[0];
- int end = *param.end[0];
- if (end < 0) end += ishape[0];
- int indptr_len = end - begin + 1;
+ CHECK((std::is_same::value)) << "SliceDimOneCsrImpl is only
implemented for CPU";
+ nnvm::dim_t begin_row = begin[0];
+ nnvm::dim_t end_row = end[0];
+ nnvm::dim_t indptr_len = end_row - begin_row + 1;
out.CheckAndAllocAuxData(kIndPtr, Shape1(indptr_len));
- if (!in.storage_initialized()) {
-out.set_aux_shape(kIndPtr, Shape1(0));
-return;
- }
// assume idx indptr share the same type
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIndPtr), RType, {
MSHADOW_IDX_TYPE_SWITCH(in.aux_type(kIdx), IType, {
MSHADOW_TYPE_SWITCH(in.dtype(), DType, {
-auto in_indptr = in.aux_data(kIndPtr).dptr();
-auto out_indptr = out.aux_data(kIndPtr).dptr();
-SliceCsrIndPtrImpl(begin, end, ctx.run_ctx, in_indptr,
out_indptr);
+RType* in_indptr = in.aux_data(kIndPtr).dptr();
+RType* out_indptr = out.aux_data(kIndPtr).dptr();
+SliceCsrIndPtrImpl(begin_row, end_row, ctx.run_ctx,
in_indptr, out_indptr);
// retrieve nnz (CPU implementation)
int nnz = out_indptr[indptr_len - 1];
+// return csr zeros if nnz = 0
+if (nnz == 0) {
+ out.set_aux_shape(kIdx, Shape1(0));
Review comment:
According to https://github.com/apache/incubator-mxnet/pull/7935 setting the
aux shape is not sufficient to generate a valid CSR ndarray. Let's always call
`ZerosCsrImpl` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on issue #8554: Disable long running tests
marcoabreu commented on issue #8554: Disable long running tests URL: https://github.com/apache/incubator-mxnet/pull/8554#issuecomment-342623637 Only the crashing ones are disabled. The long running tests are classified as nightly and are of such a complexity that they should not be executed for every single PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong commented on issue #8554: Disable long running tests
piiswrong commented on issue #8554: Disable long running tests URL: https://github.com/apache/incubator-mxnet/pull/8554#issuecomment-342611101 I think its better to try to make them run faster than disabling them This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on issue #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#issuecomment-342608301 Can you please describe what the net benefits are when using the newer cmake? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149488762
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
+Set(FIRST_CUDA FALSE)
+endif()
+include(cmake/Utils.cmake)
+mxnet_option(USE_CUDA "Build with CUDA support" ON)
Review comment:
All options should be together at the top
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149490485 ## File path: cmake/Blas.cmake ## @@ -0,0 +1,43 @@ +set(mshadow_LINKER_LIBS "") Review comment: Where did this file used to be? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149489696
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
Review comment:
nit: ?set? is usually lowercase in this file
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149490617 ## File path: cmake/FirstCuda.cmake ## @@ -0,0 +1,233 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations Review comment: What did this file used to be? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149490084
##
File path: CMakeLists.txt
##
@@ -128,14 +152,20 @@ endif()
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
-if(EXISTS ${PROJECT_SOURCE_DIR}/mshadow/cmake)
- include(mshadow/cmake/mshadow.cmake)
+if(FIRST_CUDA)
Review comment:
Can there be a more descriptive name other than ?first CUDA?? What is it,
really?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] cjolivier01 commented on a change in pull request #8572: use first class cuda with cmake 3.9 and cuda9.0 support
cjolivier01 commented on a change in pull request #8572: use first class cuda
with cmake 3.9 and cuda9.0 support
URL: https://github.com/apache/incubator-mxnet/pull/8572#discussion_r149489043
##
File path: CMakeLists.txt
##
@@ -1,17 +1,41 @@
cmake_minimum_required(VERSION 3.0.2)
-project(mxnet C CXX)
+
+
+if((${CMAKE_VERSION} VERSION_GREATER "3.9.0") OR (${CMAKE_VERSION}
VERSION_EQUAL "3.9.0"))
+Set(FIRST_CUDA TRUE)
+else()
+Set(FIRST_CUDA FALSE)
+endif()
+include(cmake/Utils.cmake)
+mxnet_option(USE_CUDA "Build with CUDA support" ON)
+mxnet_option(USE_OLDCMAKECUDA "Build with old cmake cuda" OFF)
Review comment:
Can this be set automatically from the cmake version? Maybe there is a
reason it should be set manually, what is it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piiswrong closed pull request #8540: div_scalar for sparse
piiswrong closed pull request #8540: div_scalar for sparse
URL: https://github.com/apache/incubator-mxnet/pull/8540
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
index 0f0004c6bc..742c142053 100644
--- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
+++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
@@ -130,6 +130,15 @@ MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_rminus_scalar)
.add_alias("_RMinusScalar");
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_mul_scalar)
+.describe(R"doc(Multiply an array with a scalar.
+
+``_mul_scalar`` only operates on data array of input if input is sparse.
+
+For example, if input of shape (100, 100) has only 2 non zero elements,
+i.e. input.data = [5, 6], scalar = nan,
+it will result output.data = [nan, nan] instead of 1 nans.
+
+)doc" ADD_FILELINE)
.set_attr("FInferStorageType", BinaryScalarStorageType)
.set_attr("FCompute", BinaryScalarOp::Compute)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx)
@@ -143,10 +152,28 @@
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_backward_mul_scalar)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_div_scalar)
+.describe(R"doc(Divide an array with a scalar.
+
+``_div_scalar`` only operates on data array of input if input is sparse.
+
+For example, if input of shape (100, 100) has only 2 non zero elements,
+i.e. input.data = [5, 6], scalar = nan,
+it will result output.data = [nan, nan] instead of 1 nans.
+
+)doc" ADD_FILELINE)
+.set_attr("FInferStorageType", BinaryScalarStorageType)
.set_attr("FCompute", BinaryScalarOp::Compute)
-.set_attr("FGradient", ElemwiseGradUseNone{"_div_scalar"})
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx)
+.set_attr("FGradient",
ElemwiseGradUseNone{"_backward_div_scalar"})
.add_alias("_DivScalar");
+MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_backward_div_scalar)
+.set_attr("TIsBackward", true)
+.set_attr("FInferStorageType", BinaryScalarStorageType)
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
+
+
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_rdiv_scalar)
.set_attr("FCompute", BinaryScalarOp::Compute)
.set_attr("FGradient",
ElemwiseGradUseIn{"_backward_rdiv_scalar"})
diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
index afbb46b418..32e51d3990 100644
--- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
+++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
@@ -45,7 +45,12 @@ NNVM_REGISTER_OP(_backward_mul_scalar)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
NNVM_REGISTER_OP(_div_scalar)
-.set_attr("FCompute", BinaryScalarOp::Compute);
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
+
+NNVM_REGISTER_OP(_backward_div_scalar)
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
NNVM_REGISTER_OP(_rdiv_scalar)
.set_attr("FCompute", BinaryScalarOp::Compute);
diff --git a/tests/python/unittest/test_sparse_ndarray.py
b/tests/python/unittest/test_sparse_ndarray.py
index 0bbb5ffaaa..cb564906df 100644
--- a/tests/python/unittest/test_sparse_ndarray.py
+++ b/tests/python/unittest/test_sparse_ndarray.py
@@ -246,7 +246,7 @@ def check_binary(fn, stype):
check_binary(lambda x, y: x == y, stype)
-def test_sparse_nd_binary_rop():
+def test_sparse_nd_binary_scalar_op():
N = 3
def check(fn, stype):
for _ in range(N):
@@ -269,6 +269,7 @@ def check(fn, stype):
check(lambda x: 0.5 >= x, stype)
check(lambda x: 0.5 <= x, stype)
check(lambda x: 0.5 == x, stype)
+check(lambda x: x / 2, stype)
def test_sparse_nd_binary_iop():
N = 3
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-mxnet] branch master updated: div_scalar for sparse (#8540)
This is an automated email from the ASF dual-hosted git repository.
jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c64c307 div_scalar for sparse (#8540)
c64c307 is described below
commit c64c307c99712ce1d3ed2ce8fb576cd74835438c
Author: Ziyue Huang
AuthorDate: Tue Nov 7 13:35:30 2017 -0600
div_scalar for sparse (#8540)
* div_scalar for sparse
* add docs
---
.../tensor/elemwise_binary_scalar_op_basic.cc | 29 +-
.../tensor/elemwise_binary_scalar_op_basic.cu | 7 +-
tests/python/unittest/test_sparse_ndarray.py | 3 ++-
3 files changed, 36 insertions(+), 3 deletions(-)
diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
index 0f0004c..742c142 100644
--- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
+++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc
@@ -130,6 +130,15 @@ MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_rminus_scalar)
.add_alias("_RMinusScalar");
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_mul_scalar)
+.describe(R"doc(Multiply an array with a scalar.
+
+``_mul_scalar`` only operates on data array of input if input is sparse.
+
+For example, if input of shape (100, 100) has only 2 non zero elements,
+i.e. input.data = [5, 6], scalar = nan,
+it will result output.data = [nan, nan] instead of 1 nans.
+
+)doc" ADD_FILELINE)
.set_attr("FInferStorageType", BinaryScalarStorageType)
.set_attr("FCompute", BinaryScalarOp::Compute)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx)
@@ -143,10 +152,28 @@
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_backward_mul_scalar)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_div_scalar)
+.describe(R"doc(Divide an array with a scalar.
+
+``_div_scalar`` only operates on data array of input if input is sparse.
+
+For example, if input of shape (100, 100) has only 2 non zero elements,
+i.e. input.data = [5, 6], scalar = nan,
+it will result output.data = [nan, nan] instead of 1 nans.
+
+)doc" ADD_FILELINE)
+.set_attr("FInferStorageType", BinaryScalarStorageType)
.set_attr("FCompute", BinaryScalarOp::Compute)
-.set_attr("FGradient", ElemwiseGradUseNone{"_div_scalar"})
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx)
+.set_attr("FGradient",
ElemwiseGradUseNone{"_backward_div_scalar"})
.add_alias("_DivScalar");
+MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_backward_div_scalar)
+.set_attr("TIsBackward", true)
+.set_attr("FInferStorageType", BinaryScalarStorageType)
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
+
+
MXNET_OPERATOR_REGISTER_BINARY_SCALAR(_rdiv_scalar)
.set_attr("FCompute", BinaryScalarOp::Compute)
.set_attr("FGradient",
ElemwiseGradUseIn{"_backward_rdiv_scalar"})
diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
index afbb46b..32e51d3 100644
--- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
+++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu
@@ -45,7 +45,12 @@ NNVM_REGISTER_OP(_backward_mul_scalar)
.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
NNVM_REGISTER_OP(_div_scalar)
-.set_attr("FCompute", BinaryScalarOp::Compute);
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
+
+NNVM_REGISTER_OP(_backward_div_scalar)
+.set_attr("FCompute", BinaryScalarOp::Compute)
+.set_attr("FComputeEx", BinaryScalarOp::ComputeEx);
NNVM_REGISTER_OP(_rdiv_scalar)
.set_attr("FCompute", BinaryScalarOp::Compute);
diff --git a/tests/python/unittest/test_sparse_ndarray.py
b/tests/python/unittest/test_sparse_ndarray.py
index 03a4307..34411d6 100644
--- a/tests/python/unittest/test_sparse_ndarray.py
+++ b/tests/python/unittest/test_sparse_ndarray.py
@@ -246,7 +246,7 @@ def test_sparse_nd_binary():
check_binary(lambda x, y: x == y, stype)
-def test_sparse_nd_binary_rop():
+def test_sparse_nd_binary_scalar_op():
N = 3
def check(fn, stype):
for _ in range(N):
@@ -269,6 +269,7 @@ def test_sparse_nd_binary_rop():
check(lambda x: 0.5 >= x, stype)
check(lambda x: 0.5 <= x, stype)
check(lambda x: 0.5 == x, stype)
+check(lambda x: x / 2, stype)
def test_sparse_nd_binary_iop():
N = 3
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" '].
[GitHub] piiswrong closed pull request #8550: csr ndarray iter with shuffle
piiswrong closed pull request #8550: csr ndarray iter with shuffle
URL: https://github.com/apache/incubator-mxnet/pull/8550
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/python/mxnet/io.py b/python/mxnet/io.py
index 10a12d43e5..ef2f647eec 100644
--- a/python/mxnet/io.py
+++ b/python/mxnet/io.py
@@ -35,6 +35,7 @@
from .base import check_call, build_param_doc as _build_param_doc
from .ndarray import NDArray
from .ndarray.sparse import CSRNDArray
+from .ndarray.sparse import array as sparse_array
from .ndarray import _ndarray_cls
from .ndarray import array
from .ndarray import concatenate
@@ -527,6 +528,20 @@ def _has_instance(data, dtype):
return True
return False
+def _shuffle(data, idx):
+"""Shuffle the data."""
+shuffle_data = []
+
+for k, v in data:
+if (isinstance(v, h5py.Dataset) if h5py else False):
+shuffle_data.append((k, v))
+elif isinstance(v, CSRNDArray):
+shuffle_data.append((k, sparse_array(v.asscipy()[idx], v.context)))
+else:
+shuffle_data.append((k, array(v.asnumpy()[idx], v.context)))
+
+return shuffle_data
+
class NDArrayIter(DataIter):
"""Returns an iterator for ``mx.nd.NDArray``, ``numpy.ndarray``,
``h5py.Dataset``
or ``mx.nd.sparse.CSRNDArray``.
@@ -592,8 +607,8 @@ class NDArrayIter(DataIter):
>>> label = {'label1':np.zeros(shape=(10,1)),
'label2':np.zeros(shape=(20,1))}
>>> dataiter = mx.io.NDArrayIter(data, label, 3, True,
last_batch_handle='discard')
-`NDArrayIter` also supports ``mx.nd.sparse.CSRNDArray`` with `shuffle` set
to `False`
-and `last_batch_handle` set to `discard`.
+`NDArrayIter` also supports ``mx.nd.sparse.CSRNDArray``
+with `last_batch_handle` set to `discard`.
>>> csr_data = mx.nd.array(np.arange(40).reshape((10,4))).tostype('csr')
>>> labels = np.ones([10, 1])
@@ -630,10 +645,9 @@ def __init__(self, data, label=None, batch_size=1,
shuffle=False,
super(NDArrayIter, self).__init__(batch_size)
if ((_has_instance(data, CSRNDArray) or _has_instance(label,
CSRNDArray)) and
-(shuffle or last_batch_handle != 'discard')):
+(last_batch_handle != 'discard')):
raise NotImplementedError("`NDArrayIter` only supports
``CSRNDArray``" \
- " with `shuffle` set to `False`" \
- " and `last_batch_handle` set to
`discard`.")
+ " with `last_batch_handle` set to
`discard`.")
self.data = _init_data(data, allow_empty=False, default_name=data_name)
self.label = _init_data(label, allow_empty=True,
default_name=label_name)
@@ -641,14 +655,8 @@ def __init__(self, data, label=None, batch_size=1,
shuffle=False,
# shuffle data
if shuffle:
np.random.shuffle(self.idx)
-self.data = [(k, array(v.asnumpy()[self.idx], v.context))
- if not (isinstance(v, h5py.Dataset)
- if h5py else False) else (k, v)
- for k, v in self.data]
-self.label = [(k, array(v.asnumpy()[self.idx], v.context))
- if not (isinstance(v, h5py.Dataset)
- if h5py else False) else (k, v)
- for k, v in self.label]
+self.data = _shuffle(self.data, self.idx)
+self.label = _shuffle(self.label, self.idx)
# batching
if last_batch_handle == 'discard':
diff --git a/tests/python/unittest/test_io.py b/tests/python/unittest/test_io.py
index f745d355db..2c70358397 100644
--- a/tests/python/unittest/test_io.py
+++ b/tests/python/unittest/test_io.py
@@ -27,7 +27,7 @@
except ImportError:
h5py = None
import sys
-from common import get_data
+from common import get_data, assertRaises
import unittest
@@ -161,13 +161,19 @@ def test_NDArrayIter_csr():
shape = (num_rows, num_cols)
csr, _ = rand_sparse_ndarray(shape, 'csr')
dns = csr.asnumpy()
-#test CSRNDArray with shuffle=True will throw NotImplementedError
-try:
-csr_iter = mx.io.NDArrayIter({'data': csr}, dns, batch_size,
shuffle=True,
- last_batch_handle='discard')
-assert(False)
-except NotImplementedError:
-pass
+
+# CSRNDArray with last_batch_handle not equal to 'discard' will throw
NotImplementedError
+assertRaises(NotImplementedError, mx.io.NDArrayIter, {'data': csr}, dns,
batch_size,
+ last_batch_handle='pad')
+
+# CSRNDArray with shuffle
+csr_iter = iter(mx.io.NDArrayIter({'csr_data': csr, 'dns_data': dns},
[incubator-mxnet] branch master updated: csr ndarray iter with shuffle (#8550)
This is an automated email from the ASF dual-hosted git repository.
jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 938c25f csr ndarray iter with shuffle (#8550)
938c25f is described below
commit 938c25f53d690f8b0b73f1a59c7ac274671d6e0e
Author: Ziyue Huang
AuthorDate: Tue Nov 7 13:30:01 2017 -0600
csr ndarray iter with shuffle (#8550)
* csr ndarray iter with shuffle
* address comments
---
python/mxnet/io.py | 34 +-
tests/python/unittest/test_io.py | 22 ++
2 files changed, 35 insertions(+), 21 deletions(-)
diff --git a/python/mxnet/io.py b/python/mxnet/io.py
index 10a12d4..ef2f647 100644
--- a/python/mxnet/io.py
+++ b/python/mxnet/io.py
@@ -35,6 +35,7 @@ from .base import mx_real_t
from .base import check_call, build_param_doc as _build_param_doc
from .ndarray import NDArray
from .ndarray.sparse import CSRNDArray
+from .ndarray.sparse import array as sparse_array
from .ndarray import _ndarray_cls
from .ndarray import array
from .ndarray import concatenate
@@ -527,6 +528,20 @@ def _has_instance(data, dtype):
return True
return False
+def _shuffle(data, idx):
+"""Shuffle the data."""
+shuffle_data = []
+
+for k, v in data:
+if (isinstance(v, h5py.Dataset) if h5py else False):
+shuffle_data.append((k, v))
+elif isinstance(v, CSRNDArray):
+shuffle_data.append((k, sparse_array(v.asscipy()[idx], v.context)))
+else:
+shuffle_data.append((k, array(v.asnumpy()[idx], v.context)))
+
+return shuffle_data
+
class NDArrayIter(DataIter):
"""Returns an iterator for ``mx.nd.NDArray``, ``numpy.ndarray``,
``h5py.Dataset``
or ``mx.nd.sparse.CSRNDArray``.
@@ -592,8 +607,8 @@ class NDArrayIter(DataIter):
>>> label = {'label1':np.zeros(shape=(10,1)),
'label2':np.zeros(shape=(20,1))}
>>> dataiter = mx.io.NDArrayIter(data, label, 3, True,
last_batch_handle='discard')
-`NDArrayIter` also supports ``mx.nd.sparse.CSRNDArray`` with `shuffle` set
to `False`
-and `last_batch_handle` set to `discard`.
+`NDArrayIter` also supports ``mx.nd.sparse.CSRNDArray``
+with `last_batch_handle` set to `discard`.
>>> csr_data = mx.nd.array(np.arange(40).reshape((10,4))).tostype('csr')
>>> labels = np.ones([10, 1])
@@ -630,10 +645,9 @@ class NDArrayIter(DataIter):
super(NDArrayIter, self).__init__(batch_size)
if ((_has_instance(data, CSRNDArray) or _has_instance(label,
CSRNDArray)) and
-(shuffle or last_batch_handle != 'discard')):
+(last_batch_handle != 'discard')):
raise NotImplementedError("`NDArrayIter` only supports
``CSRNDArray``" \
- " with `shuffle` set to `False`" \
- " and `last_batch_handle` set to
`discard`.")
+ " with `last_batch_handle` set to
`discard`.")
self.data = _init_data(data, allow_empty=False, default_name=data_name)
self.label = _init_data(label, allow_empty=True,
default_name=label_name)
@@ -641,14 +655,8 @@ class NDArrayIter(DataIter):
# shuffle data
if shuffle:
np.random.shuffle(self.idx)
-self.data = [(k, array(v.asnumpy()[self.idx], v.context))
- if not (isinstance(v, h5py.Dataset)
- if h5py else False) else (k, v)
- for k, v in self.data]
-self.label = [(k, array(v.asnumpy()[self.idx], v.context))
- if not (isinstance(v, h5py.Dataset)
- if h5py else False) else (k, v)
- for k, v in self.label]
+self.data = _shuffle(self.data, self.idx)
+self.label = _shuffle(self.label, self.idx)
# batching
if last_batch_handle == 'discard':
diff --git a/tests/python/unittest/test_io.py b/tests/python/unittest/test_io.py
index f745d35..2c70358 100644
--- a/tests/python/unittest/test_io.py
+++ b/tests/python/unittest/test_io.py
@@ -27,7 +27,7 @@ try:
except ImportError:
h5py = None
import sys
-from common import get_data
+from common import get_data, assertRaises
import unittest
@@ -161,13 +161,19 @@ def test_NDArrayIter_csr():
shape = (num_rows, num_cols)
csr, _ = rand_sparse_ndarray(shape, 'csr')
dns = csr.asnumpy()
-#test CSRNDArray with shuffle=True will throw NotImplementedError
-try:
-csr_iter = mx.io.NDArrayIter({'data': csr}, dns, batch_size,
shuffle=True,
- last_batch_handle='discard')
-assert(False)
-except NotImplementedError:
-pass
+
+# CSRNDArray with last_batch_handle not equal
[GitHub] piiswrong commented on issue #8578: Add development instructions and update installation instructions
piiswrong commented on issue #8578: Add development instructions and update installation instructions URL: https://github.com/apache/incubator-mxnet/pull/8578#issuecomment-342594203 why add a new page instead of putting it into build from source section? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
