piiswrong commented on a change in pull request #10628: [MXNET-342] Fix the
multi worker Dataloader
URL: https://github.com/apache/incubator-mxnet/pull/10628#discussion_r183123741
##########
File path: python/mxnet/gluon/data/dataset.py
##########
@@ -173,8 +173,15 @@ class RecordFileDataset(Dataset):
Path to rec file.
"""
def __init__(self, filename):
- idx_file = os.path.splitext(filename)[0] + '.idx'
- self._record = recordio.MXIndexedRecordIO(idx_file, filename, 'r')
+ self._filename = filename
+ self.reload_recordfile()
+
+ def reload_recordfile(self):
+ """
+ Reload the record file.
+ """
+ idx_file = os.path.splitext(self._filename)[0] + '.idx'
+ self._record = recordio.MXIndexedRecordIO(idx_file, self._filename,
'r')
Review comment:
we should fix this in backend. Why would forking cause a problem? File
descriptors should be duplicated when forking
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
