leezu opened a new issue #18618:
URL: https://github.com/apache/incubator-mxnet/issues/18618
```
[2020-06-24T20:40:36.419Z] =================================== FAILURES
===================================
[2020-06-24T20:40:36.419Z] _________________________
test_contrib_DataLoaderIter __________________________
[2020-06-24T20:40:36.419Z] [gw0] linux -- Python 3.6.9
/opt/rh/rh-python36/root/usr/bin/python3
[2020-06-24T20:40:36.419Z]
[2020-06-24T20:40:36.419Z] def test_contrib_DataLoaderIter():
[2020-06-24T20:40:36.419Z] def test_mnist_batches(batch_size,
expected, last_batch='discard'):
[2020-06-24T20:40:36.419Z] dataset = MNIST(train=False)
[2020-06-24T20:40:36.419Z] dataloader = DataLoader(dataset,
batch_size, last_batch=last_batch)
[2020-06-24T20:40:36.419Z] test_iter = DataLoaderIter(dataloader)
[2020-06-24T20:40:36.419Z] batch = next(test_iter)
[2020-06-24T20:40:36.419Z] assert batch.data[0].shape ==
(batch_size, 28, 28, 1)
[2020-06-24T20:40:36.419Z] assert batch.label[0].shape ==
(batch_size,)
[2020-06-24T20:40:36.419Z] count = 0
[2020-06-24T20:40:36.419Z] test_iter.reset()
[2020-06-24T20:40:36.419Z] for batch in test_iter:
[2020-06-24T20:40:36.419Z] count += 1
[2020-06-24T20:40:36.419Z] assert count == expected, "expected
{} batches, given {}".format(expected, count)
[2020-06-24T20:40:36.419Z]
[2020-06-24T20:40:36.419Z] num_examples = 10000
[2020-06-24T20:40:36.419Z] test_mnist_batches(50, num_examples //
50, 'discard')
[2020-06-24T20:40:36.419Z] test_mnist_batches(31, num_examples //
31, 'discard')
[2020-06-24T20:40:36.419Z] > test_mnist_batches(31, num_examples //
31, 'rollover')
[2020-06-24T20:40:36.419Z]
[2020-06-24T20:40:36.419Z] tests/python/unittest/test_contrib_io.py:41:
[2020-06-24T20:40:36.419Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-06-24T20:40:36.419Z] tests/python/unittest/test_contrib_io.py:34: in
test_mnist_batches
[2020-06-24T20:40:36.419Z] for batch in test_iter:
[2020-06-24T20:40:36.419Z] python/mxnet/io/io.py:229: in __next__
[2020-06-24T20:40:36.419Z] return self.next()
[2020-06-24T20:40:36.419Z] python/mxnet/io/io.py:222: in next
[2020-06-24T20:40:36.419Z] if self.iter_next():
[2020-06-24T20:40:36.419Z] python/mxnet/contrib/io.py:69: in iter_next
[2020-06-24T20:40:36.419Z] self._current_batch = next(self._iter)
[2020-06-24T20:40:36.419Z] python/mxnet/gluon/data/dataloader.py:659: in
same_process_iter
[2020-06-24T20:40:36.419Z] ret = self._batchify_fn([self._dataset[idx]
for idx in batch])
[2020-06-24T20:40:36.419Z] python/mxnet/gluon/data/batchify.py:91: in
__call__
[2020-06-24T20:40:36.419Z] return [self.__call__(i) for i in data]
[2020-06-24T20:40:36.419Z] python/mxnet/gluon/data/batchify.py:91: in
<listcomp>
[2020-06-24T20:40:36.420Z] return [self.__call__(i) for i in data]
[2020-06-24T20:40:36.420Z] python/mxnet/gluon/data/batchify.py:98: in
__call__
[2020-06-24T20:40:36.420Z] return _arr.array(out, dtype=dtype)
[2020-06-24T20:40:36.420Z] python/mxnet/ndarray/utils.py:146: in array
[2020-06-24T20:40:36.420Z] return _array(source_array, ctx=ctx,
dtype=dtype)
[2020-06-24T20:40:36.420Z] python/mxnet/ndarray/ndarray.py:3376: in array
[2020-06-24T20:40:36.420Z] arr[:] = source_array
[2020-06-24T20:40:36.420Z] python/mxnet/ndarray/ndarray.py:550: in
__setitem__
[2020-06-24T20:40:36.420Z] self._set_nd_basic_indexing(key, value)
[2020-06-24T20:40:36.420Z] python/mxnet/ndarray/ndarray.py:993: in
_set_nd_basic_indexing
[2020-06-24T20:40:36.420Z] self._sync_copyfrom(value)
[2020-06-24T20:40:36.420Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] self =
[2020-06-24T20:40:36.420Z] [4 7 0 9 0 0 3 7 9 3 0 2 0 1 0 1 0 4 0 1 0 4 7 9
6 2 6 2 2 9 9]
[2020-06-24T20:40:36.420Z] <NDArray 31 @cpu(0)>
[2020-06-24T20:40:36.420Z] source_array = array([4, 7, 0, 9, 0, 0, 3, 7, 9,
3, 0, 2, 0, 1, 0, 1, 0, 4, 0, 1, 0, 4,
[2020-06-24T20:40:36.420Z] 7, 9, 6, 2, 6, 2, 2, 9, 9], dtype=int32)
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] def _sync_copyfrom(self, source_array):
[2020-06-24T20:40:36.420Z] """Performs a synchronized copy from the
`source_array` to the current array.
[2020-06-24T20:40:36.420Z] This is called through ``x[:] =
source_array``, where the `source_array`
[2020-06-24T20:40:36.420Z] is a `numpy.ndarray` or array-like object.
[2020-06-24T20:40:36.420Z] This function blocks until all the
pending read/write operations with respect
[2020-06-24T20:40:36.420Z] to the current `NDArray` are finished and
carry out the copy operation to the
[2020-06-24T20:40:36.420Z] current NDArray.
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] Parameters
[2020-06-24T20:40:36.420Z] ----------
[2020-06-24T20:40:36.420Z] source_array : array_like
[2020-06-24T20:40:36.420Z] The data source we would like to copy
from.
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] Example
[2020-06-24T20:40:36.420Z] -------
[2020-06-24T20:40:36.420Z] >>> a = mx.nd.array([1, 2])
[2020-06-24T20:40:36.420Z] >>> a.asnumpy()
[2020-06-24T20:40:36.420Z] array([ 1., 2.], dtype=float32)
[2020-06-24T20:40:36.420Z] >>> a[:] = np.array([3, 4])
[2020-06-24T20:40:36.420Z] >> a.asnumpy()
[2020-06-24T20:40:36.420Z] array([ 3., 4.], dtype=float32)
[2020-06-24T20:40:36.420Z] """
[2020-06-24T20:40:36.420Z] if not isinstance(source_array,
np.ndarray):
[2020-06-24T20:40:36.420Z] try:
[2020-06-24T20:40:36.420Z] source_array =
np.array(source_array, dtype=self.dtype)
[2020-06-24T20:40:36.420Z] except:
[2020-06-24T20:40:36.420Z] raise TypeError('array must
consist of array-like data,' +
[2020-06-24T20:40:36.420Z] 'type %s is not
supported' % str(type(array)))
[2020-06-24T20:40:36.420Z] source_array = np.asarray(source_array,
dtype=self.dtype, order='C')
[2020-06-24T20:40:36.420Z] if source_array.shape != self.shape:
[2020-06-24T20:40:36.420Z] raise ValueError('Shape inconsistent:
expected %s vs got %s'%(
[2020-06-24T20:40:36.420Z] str(source_array.shape),
str(self.shape)))
[2020-06-24T20:40:36.420Z] check_call(_LIB.MXNDArraySyncCopyFromCPU(
[2020-06-24T20:40:36.420Z] self.handle,
[2020-06-24T20:40:36.420Z]
source_array.ctypes.data_as(ctypes.c_void_p),
[2020-06-24T20:40:36.420Z] > ctypes.c_size_t(source_array.size)))
[2020-06-24T20:40:36.420Z] E Failed: Timeout >1200.0s
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] python/mxnet/ndarray/ndarray.py:1333: Failed
[2020-06-24T20:40:36.420Z] ---------------------------- Captured stderr
setup -----------------------------
[2020-06-24T20:40:36.420Z] DEBUG:root:np/mx/python random seeds are set to
323026922, use MXNET_TEST_SEED=323026922 to reproduce.
[2020-06-24T20:40:36.420Z] ------------------------------ Captured log setup
------------------------------
[2020-06-24T20:40:36.420Z] DEBUG root:conftest.py:193 np/mx/python random
seeds are set to 323026922, use MXNET_TEST_SEED=323026922 to reproduce.
[2020-06-24T20:40:36.420Z] ----------------------------- Captured stdout
call -----------------------------
[2020-06-24T20:40:36.420Z] Downloading
tests/data/datasets/mnist/t10k-images-idx3-ubyte.gz from
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/mnist/t10k-images-idx3-ubyte.gz...
[2020-06-24T20:40:36.420Z] Downloading
tests/data/datasets/mnist/t10k-labels-idx1-ubyte.gz from
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/mnist/t10k-labels-idx1-ubyte.gz...
[2020-06-24T20:40:36.420Z] ----------------------------- Captured stderr
call -----------------------------
[2020-06-24T20:40:36.420Z] DEBUG:urllib3.connectionpool:Starting new HTTPS
connection (1): apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
[2020-06-24T20:40:36.420Z]
DEBUG:urllib3.connectionpool:https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
"GET /gluon/dataset/mnist/t10k-images-idx3-ubyte.gz HTTP/1.1" 200 1648877
[2020-06-24T20:40:36.420Z] DEBUG:urllib3.connectionpool:Starting new HTTPS
connection (1): apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
[2020-06-24T20:40:36.420Z]
DEBUG:urllib3.connectionpool:https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
"GET /gluon/dataset/mnist/t10k-labels-idx1-ubyte.gz HTTP/1.1" 200 4542
[2020-06-24T20:40:36.420Z]
[2020-06-24T20:40:36.420Z] ~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown>
(140259838953216) ~~~~~~~~~~~~~~~~~~~~~
[2020-06-24T20:40:36.420Z] File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/execnet/gateway_base.py",
line 285, in _perform_spawn
[2020-06-24T20:40:36.420Z] reply.run()
[2020-06-24T20:40:36.420Z] File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/execnet/gateway_base.py",
line 220, in run
[2020-06-24T20:40:36.420Z] self._result = func(*args, **kwargs)
[2020-06-24T20:40:36.420Z] File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/execnet/gateway_base.py",
line 967, in _thread_receiver
[2020-06-24T20:40:36.420Z] msg = Message.from_io(io)
[2020-06-24T20:40:36.420Z] File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/execnet/gateway_base.py",
line 432, in from_io
[2020-06-24T20:40:36.420Z] header = io.read(9) # type 1, channel 4,
payload 4
[2020-06-24T20:40:36.420Z] File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/execnet/gateway_base.py",
line 400, in read
[2020-06-24T20:40:36.420Z] data = self._read(numbytes - len(buf))
[2020-06-24T20:40:36.420Z] ------------------------------ Captured log call
-------------------------------
[2020-06-24T20:40:36.420Z] DEBUG
urllib3.connectionpool:connectionpool.py:941 Starting new HTTPS connection (1):
apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
[2020-06-24T20:40:36.420Z] DEBUG
urllib3.connectionpool:connectionpool.py:442
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443 "GET
/gluon/dataset/mnist/t10k-images-idx3-ubyte.gz HTTP/1.1" 200 1648877
[2020-06-24T20:40:36.420Z] DEBUG
urllib3.connectionpool:connectionpool.py:941 Starting new HTTPS connection (1):
apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443
[2020-06-24T20:40:36.420Z] DEBUG
urllib3.connectionpool:connectionpool.py:442
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443 "GET
/gluon/dataset/mnist/t10k-labels-idx1-ubyte.gz HTTP/1.1" 200 4542
[2020-06-24T20:40:36.420Z] --------------------------- Captured stderr
teardown ---------------------------
[2020-06-24T20:40:36.420Z] INFO:root:np/mx/python random seeds are set to
323026922, use MXNET_TEST_SEED=323026922 to reproduce.
[2020-06-24T20:40:36.420Z] ---------------------------- Captured log
teardown -----------------------------
[2020-06-24T20:40:36.420Z] INFO root:conftest.py:210 np/mx/python random
seeds are set to 323026922, use MXNET_TEST_SEED=323026922 to reproduce.
```
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/centos-cpu/branches/PR-18616/runs/1/nodes/209/steps/281/log/?start=0
This happened in the CD pipeline tests. FYI cc @mseth10
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
