[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180909444 ## File path: python/mxnet/io.py ## @@ -645,8 +647,8 @@ def __init__(self, data, label=None, batch_size=1, shuffle=False, label_name='softmax_label'): super(NDArrayIter, self).__init__(batch_size) -self.data = _init_data(data, allow_empty=False, default_name=data_name) -self.label = _init_data(label, allow_empty=True, default_name=label_name) +self.data, self.renamed_data = _init_data(data, allow_empty=False, default_name=data_name) +self.label, _ = _init_data(label, allow_empty=True, default_name=label_name) Review comment: Looking in to this, i do not see a scenario in which we will have issues with the label. User can create NDIterator for labels and can bind only few of them . It should not affect the forward pass . Correct me if I am missing something here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864756 ## File path: python/mxnet/module/base_module.py ## @@ -268,6 +285,10 @@ def score(self, eval_data, eval_metric, num_batch=None, batch_end_callback=None, for callback in _as_list(score_end_callback): callback(params) +# restoring original user data if changed: +if orig_eval_data is not None: Review comment: hmm .. will have to think about this , coz we cant restore before all the forward op is completed on all the batches. How abt try/catch the forward loop? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864685 ## File path: python/mxnet/module/base_module.py ## @@ -364,6 +385,22 @@ def predict(self, eval_data, num_batch=None, merge_batches=True, reset=True, output_list = [] +# If NDArrayIter data provided is a dict. This makes sure that only relevant data items +# matching the data names, provided during bind time, are send for eval. +orig_eval_data = None +if isinstance(eval_data, io.NDArrayIter) and \ +not eval_data.renamed_data and \ +eval_data.data and \ +isinstance(eval_data.data[0], tuple): Review comment: "eval_data.data" line 393 checks for that . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175]
NdArrayiterator dict fix
URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864703
##
File path: python/mxnet/io.py
##
@@ -506,6 +507,7 @@ def _init_data(data, allow_empty, default_name):
else:
data = OrderedDict( # pylint: disable=redefined-variable-type
[('_%d_%s' % (i, default_name), d) for i, d in
enumerate(data)])
+renamed_data = True
Review comment:
will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864745 ## File path: python/mxnet/module/base_module.py ## @@ -244,6 +245,22 @@ def score(self, eval_data, eval_metric, num_batch=None, batch_end_callback=None, eval_metric.reset() actual_num_batch = 0 +# If NDArrayIter data provided is a dict. This makes sure that only relevant data items +# matching the data names, provided during bind time, are send for eval. +orig_eval_data = None +if isinstance(eval_data, io.NDArrayIter) and \ +not eval_data.renamed_data and \ +eval_data.data and \ +isinstance(eval_data.data[0], tuple): +# Keeping a copy of original data. +orig_eval_data = eval_data.data +data_dict = dict(eval_data.data) + +for k, _ in eval_data.data: +if k not in self.data_names: +del data_dict[k] +eval_data.data = list(data_dict.items()) Review comment: Did you mean create a func() ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864722 ## File path: python/mxnet/io.py ## @@ -645,8 +647,8 @@ def __init__(self, data, label=None, batch_size=1, shuffle=False, label_name='softmax_label'): super(NDArrayIter, self).__init__(batch_size) -self.data = _init_data(data, allow_empty=False, default_name=data_name) -self.label = _init_data(label, allow_empty=True, default_name=label_name) +self.data, self.renamed_data = _init_data(data, allow_empty=False, default_name=data_name) +self.label, _ = _init_data(label, allow_empty=True, default_name=label_name) Review comment: I was thinking about that too , but was not sure what would be a good place to handle labels, probably at bind . But I have to dig some more. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180864676 ## File path: python/mxnet/module/base_module.py ## @@ -364,6 +385,22 @@ def predict(self, eval_data, num_batch=None, merge_batches=True, reset=True, output_list = [] +# If NDArrayIter data provided is a dict. This makes sure that only relevant data items +# matching the data names, provided during bind time, are send for eval. +orig_eval_data = None +if isinstance(eval_data, io.NDArrayIter) and \ +not eval_data.renamed_data and \ +eval_data.data and \ +isinstance(eval_data.data[0], tuple): +# Keeping a copy of original data. +orig_eval_data = eval_data.data +data_dict = dict(eval_data.data) + +for k, _ in eval_data.data: +if k not in self.data_names: +del data_dict[k] +eval_data.data = list(data_dict.items()) Review comment: eval.data/data_dict is already sorted( io.py in _init_data()) , I am just deleting entries from there, should be already in sorted order. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180290804 ## File path: python/mxnet/io.py ## @@ -645,8 +647,8 @@ def __init__(self, data, label=None, batch_size=1, shuffle=False, label_name='softmax_label'): super(NDArrayIter, self).__init__(batch_size) -self.data = _init_data(data, allow_empty=False, default_name=data_name) -self.label = _init_data(label, allow_empty=True, default_name=label_name) +self.data, self.renamedData = _init_data(data, allow_empty=False, default_name=data_name) Review comment: done ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix
spidyDev commented on a change in pull request #10425: [MXNET-175] NdArrayiterator dict fix URL: https://github.com/apache/incubator-mxnet/pull/10425#discussion_r180290853 ## File path: python/mxnet/module/base_module.py ## @@ -363,6 +363,19 @@ def predict(self, eval_data, num_batch=None, merge_batches=True, reset=True, eval_data.reset() output_list = [] +# If data provided is a dict. This makes sure that only relevant data items Review comment: Yes, thanks for pointing this out. Added to score() too!! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
