ThomasDelteil commented on a change in pull request #10628: [MXNET-342] Fix the
multi worker Dataloader
URL: https://github.com/apache/incubator-mxnet/pull/10628#discussion_r183124801
##########
File path: python/mxnet/gluon/data/dataset.py
##########
@@ -173,8 +173,15 @@ class RecordFileDataset(Dataset):
Path to rec file.
"""
def __init__(self, filename):
- idx_file = os.path.splitext(filename)[0] + '.idx'
- self._record = recordio.MXIndexedRecordIO(idx_file, filename, 'r')
+ self._filename = filename
+ self.reload_recordfile()
+
+ def reload_recordfile(self):
+ """
+ Reload the record file.
+ """
+ idx_file = os.path.splitext(self._filename)[0] + '.idx'
+ self._record = recordio.MXIndexedRecordIO(idx_file, self._filename,
'r')
Review comment:
That would be the ideal solution indeed.
https://groups.google.com/forum/#!topic/comp.lang.python/x-C31fCSZso
contrary to what I stated earlier, It looks like the actual problem could be
that the file descriptors get closed rather than shared?
I don't see an easy way to set `close_fds=False`
https://docs.python.org/3/library/subprocess.html#popen-constructor since we
are using the multiprocessing package rather than subprocess.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
