[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182790921
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
+if kernel is not None:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT,
kernel_name=kernel)
+else:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT)
+nb, stuff = eprocessor.preprocess(notebook, {'metadata': {'path':
working_dir}})
+print(stuff)
+except Exception as err:
+err_msg = str(err)
+errors.append(err_msg)
+finally:
+if notebook is not None:
+output_file = os.path.join(working_dir, "output.txt")
+nbformat.write(notebook, output_file)
+output_nb = open(output_file, mode='r')
+for line in output_nb:
+if "Warning:" in line:
+errors.append("Warning:\n"+line)
+if len(errors) > 0:
+print('\n'.join(errors))
+return False
+return True
+
+
+
+def test_basic_ndarray():
+ assert _test_tutorial_nb('basic/ndarray')
+
+def test_basic_ndarray_indexing():
+assert _test_tutorial_nb('basic/ndarray_indexing')
+
+def test_basic_symbol():
+assert _test_tutorial_nb('basic/symbol')
+
+def test_basic_module():
+assert _test_tutorial_nb('basic/module')
+
+def test_basic_data():
+assert _test_tutorial_nb('basic/data')
+
+def test_gluon_customop():
+assert _test_tutorial_nb('gluon/customop')
+
+def test_gluon_data_augmentation():
+assert _test_tutorial_nb('gluon/data_augmentation')
+
+def test_gluon_datasets():
+assert True
+# Investigating flakiness with docker
+#assert _test_tutorial_nb('gluon/datasets')
Review comment:
That is a weird one, might leave out for this iteration and investigate more
next week. It runs fine outside the container and in the notebook but hangs
indefinitely when run through nosetest in a container using up all GPUs
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182790921
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
+if kernel is not None:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT,
kernel_name=kernel)
+else:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT)
+nb, stuff = eprocessor.preprocess(notebook, {'metadata': {'path':
working_dir}})
+print(stuff)
+except Exception as err:
+err_msg = str(err)
+errors.append(err_msg)
+finally:
+if notebook is not None:
+output_file = os.path.join(working_dir, "output.txt")
+nbformat.write(notebook, output_file)
+output_nb = open(output_file, mode='r')
+for line in output_nb:
+if "Warning:" in line:
+errors.append("Warning:\n"+line)
+if len(errors) > 0:
+print('\n'.join(errors))
+return False
+return True
+
+
+
+def test_basic_ndarray():
+ assert _test_tutorial_nb('basic/ndarray')
+
+def test_basic_ndarray_indexing():
+assert _test_tutorial_nb('basic/ndarray_indexing')
+
+def test_basic_symbol():
+assert _test_tutorial_nb('basic/symbol')
+
+def test_basic_module():
+assert _test_tutorial_nb('basic/module')
+
+def test_basic_data():
+assert _test_tutorial_nb('basic/data')
+
+def test_gluon_customop():
+assert _test_tutorial_nb('gluon/customop')
+
+def test_gluon_data_augmentation():
+assert _test_tutorial_nb('gluon/data_augmentation')
+
+def test_gluon_datasets():
+assert True
+# Investigating flakiness with docker
+#assert _test_tutorial_nb('gluon/datasets')
Review comment:
That is a weird one, might leave out for this iteration and investigate more
next week. It runs fine outside the container and in the notebook but hangs
indefinitely when run through nosetest in a container using up all CPUs
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182790921
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
+if kernel is not None:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT,
kernel_name=kernel)
+else:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT)
+nb, stuff = eprocessor.preprocess(notebook, {'metadata': {'path':
working_dir}})
+print(stuff)
+except Exception as err:
+err_msg = str(err)
+errors.append(err_msg)
+finally:
+if notebook is not None:
+output_file = os.path.join(working_dir, "output.txt")
+nbformat.write(notebook, output_file)
+output_nb = open(output_file, mode='r')
+for line in output_nb:
+if "Warning:" in line:
+errors.append("Warning:\n"+line)
+if len(errors) > 0:
+print('\n'.join(errors))
+return False
+return True
+
+
+
+def test_basic_ndarray():
+ assert _test_tutorial_nb('basic/ndarray')
+
+def test_basic_ndarray_indexing():
+assert _test_tutorial_nb('basic/ndarray_indexing')
+
+def test_basic_symbol():
+assert _test_tutorial_nb('basic/symbol')
+
+def test_basic_module():
+assert _test_tutorial_nb('basic/module')
+
+def test_basic_data():
+assert _test_tutorial_nb('basic/data')
+
+def test_gluon_customop():
+assert _test_tutorial_nb('gluon/customop')
+
+def test_gluon_data_augmentation():
+assert _test_tutorial_nb('gluon/data_augmentation')
+
+def test_gluon_datasets():
+assert True
+# Investigating flakiness with docker
+#assert _test_tutorial_nb('gluon/datasets')
Review comment:
That is a weird one, might leave out for this iteration and investigate more
next week. It runs fine outside the container and in the notebook but hangs
indefinitely when run through nosetest in a container.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182790384 ## File path: tests/tutorials/test_sanity_tutorials.py ## @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import glob +import os +import re + +# White list of non-downloadable tutorials Review comment: That's the tests we are ok for them not being downloadable as jupyter notebooks, typically C++, Scala, R tutorials This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182789983
##
File path: docs/mxdoc.py
##
@@ -367,7 +367,8 @@ def add_buttons(app, docname, source):
# source[i] = '\n'.join(lines)
def setup(app):
-app.connect("builder-inited", build_mxnet)
+if os.getenv('MXNET_DOCS_BUILD_MXNET', '1') == '1':
Review comment:
will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182789576
##
File path: ci/docker/runtime_functions.sh
##
@@ -385,6 +386,28 @@ unittest_ubuntu_python2_gpu() {
nosetests-2.7 --verbose tests/python/gpu
}
+tutorialtest_ubuntu_python3_gpu() {
+set -ex
+cd /work/mxnet/docs
+export MXNET_DOCS_BUILD_MXNET=0
+make html
+export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TUTORIAL_TEST_KERNEL=python3
+cd /work/mxnet/tests/tutorials && nosetests-3.4 test_tutorials.py
--nologcapture
+}
+
+tutorialtest_ubuntu_python2_gpu() {
+set -ex
+cd /work/mxnet/docs
+export MXNET_DOCS_BUILD_MXNET=0
+make html
+export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TUTORIAL_TEST_KERNEL=python2
+cd /work/mxnet/tests/tutorials && nosetests-3.4 test_tutorials.py
--nologcapture
Review comment:
nosetest is trying to be smart but in this case since we are spawning a new
process we get a whole load of unwanted extra logging:
- the raw text of the notebook
- the debug print out of jupyter (e.g the heartbeat signals of the kernels)
I am already printing the errors and warning (which are captured even with
--nologcapture)
I use nosetest-3.4 to run the test as it supports unicode by default,
however `MXNET_TUTORIAL_TEST_KERNEL=python2` is what controls the environment
the notebooks are being run
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182789576
##
File path: ci/docker/runtime_functions.sh
##
@@ -385,6 +386,28 @@ unittest_ubuntu_python2_gpu() {
nosetests-2.7 --verbose tests/python/gpu
}
+tutorialtest_ubuntu_python3_gpu() {
+set -ex
+cd /work/mxnet/docs
+export MXNET_DOCS_BUILD_MXNET=0
+make html
+export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TUTORIAL_TEST_KERNEL=python3
+cd /work/mxnet/tests/tutorials && nosetests-3.4 test_tutorials.py
--nologcapture
+}
+
+tutorialtest_ubuntu_python2_gpu() {
+set -ex
+cd /work/mxnet/docs
+export MXNET_DOCS_BUILD_MXNET=0
+make html
+export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TUTORIAL_TEST_KERNEL=python2
+cd /work/mxnet/tests/tutorials && nosetests-3.4 test_tutorials.py
--nologcapture
Review comment:
nosetest is trying to be smart but in this case since we are spawning a new
process we get a whole load of unwanted extra logging:
- the raw text of the notebook
- the debug print out of jupyter (e.g the heartbeat signals of the kernels)
I am already printing the errors and warning (which are captured even with
--nologcapture)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182787563
##
File path: ci/docker/runtime_functions.sh
##
@@ -349,6 +349,7 @@ sanity_check() {
tools/license_header.py check
make cpplint rcpplint jnilint
make pylint
+nosetests-3.4 tests/tutorials/test_sanity_tutorials.py
Review comment:
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182786851
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
+if kernel is not None:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT,
kernel_name=kernel)
+else:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT)
+nb, stuff = eprocessor.preprocess(notebook, {'metadata': {'path':
working_dir}})
+print(stuff)
+except Exception as err:
+err_msg = str(err)
+errors.append(err_msg)
+finally:
+if notebook is not None:
+output_file = os.path.join(working_dir, "output.txt")
Review comment:
The filepath is unique per tutorial, it actually supports parallelization
pretty well.
Parallelization with nose test is much simpler than we discussed last time,
just add:
`--processes=8 --process-timeout=1800 (--process-restartworker)`
It does work with this test, reducing the time necessary to run them by 60%
with 8 workers. However, I am not sure how mature this nosetest plugin is. I
think the fact these tests spawn an extra process confuses nosetest and I have
witnessed some nostests workers crashing that leave python processes orphans,
using up GPU memory, and on top of that that particular test case was
considered successful.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182784955
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
Review comment:
will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182784893
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
Review comment:
will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182784791
##
File path: tests/tutorials/test_tutorials.py
##
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#pylint: disable=no-member, too-many-locals, too-many-branches, no-self-use,
broad-except, lost-exception, too-many-nested-blocks, too-few-public-methods,
invalid-name
+"""
+This script converts all python tutorials into python script
+and tests whether there is any warning or error.
+After running python script, it will also convert markdown files
+to notebooks to make sure notebook execution has no error.
+"""
+import os
+import warnings
+import imp
+import shutil
+import time
+import argparse
+import traceback
+import nbformat
+from nbconvert.preprocessors import ExecutePreprocessor
+import sys
+
+
+TIME_OUT = 1800
+temp_dir = 'tmp_notebook'
+
+def _test_tutorial_nb(tutorial):
+"""Run tutorial jupyter notebook to catch any execution error.
+
+Parameters
+--
+tutorial : str
+tutorial name in folder/tutorial format
+"""
+
+tutorial_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'docs',
'_build', 'html', 'tutorials')
+tutorial_path = os.path.join(*([tutorial_dir] + tutorial.split('/')))
+
+kernel = os.getenv('MXNET_TUTORIAL_TEST_KERNEL', None)
+no_cache = os.getenv('MXNET_TUTORIAL_TEST_NO_CACHE', False)
+
+working_dir = os.path.join(*([temp_dir] + tutorial.split('/')))
+
+if no_cache:
+print("Cleaning and setting up temp directory
'{}'".format(working_dir))
+shutil.rmtree(temp_dir, ignore_errors=True)
+
+errors = []
+notebook = None
+if not os.path.isdir(working_dir):
+os.makedirs(working_dir)
+try:
+notebook = nbformat.read(tutorial_path + '.ipynb', as_version=4)
+if kernel is not None:
+eprocessor = ExecutePreprocessor(timeout=TIME_OUT,
kernel_name=kernel)
Review comment:
that is the timeout for the execution of a single notebook. Good point, I am
not sure whether that would fail the test or just move on.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182784310 ## File path: ci/docker/install/ubuntu_tutorials.sh ## @@ -0,0 +1,26 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# build and install are separated so changes to build don't invalidate +# the whole docker cache for the image + +set -ex +apt-get install graphviz python-opencv +pip2 install jupyter matplotlib Pillow opencv-python scipy scikit-learn h5py==2.8.0rc1 graphviz +pip3 install jupyter matplotlib Pillow opencv-python scipy scikit-learn h5py==2.8.0rc1 graphviz Review comment: ok will do This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182784149 ## File path: ci/docker/install/ubuntu_scala.sh ## @@ -23,9 +23,8 @@ set -ex # install libraries for mxnet's scala package on ubuntu apt-get install -y software-properties-common -add-apt-repository -y ppa:webupd8team/java apt-get update -echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections -apt-get install -y oracle-java8-installer -apt-get install -y oracle-java8-set-default -apt-get update && apt-get install -y maven \ No newline at end of file +sleep $[ ( $RANDOM % 10 ) + 1 ]s Review comment: happy to do it, however I noticed a lot less failed image build after adding that. I wonder if the fact that we send 20 requests at the same time for the same package might be a problem. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add
tutorial tests to the CI
URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182783772
##
File path: ci/build.py
##
@@ -157,6 +163,11 @@ def script_name() -> str:
help="Use nvidia docker",
action='store_true')
+parser.add_argument("--shm-size",
+help="Size of the shared memory allocated for the
container (e.g '1g')",
Review comment:
That's the size of the /dev/shm inside the container, which is used for
interprocess communication for `DataLoader`s. By default it was only 50m, which
causes hang ups when used for real-world example like in the tutorials, with 32
workers. https://docs.docker.com/engine/reference/commandline/run/
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI
ThomasDelteil commented on a change in pull request #10608: [MXNET-292] Add tutorial tests to the CI URL: https://github.com/apache/incubator-mxnet/pull/10608#discussion_r182782147 ## File path: ci/docker/Dockerfile.build.ubuntu_gpu ## @@ -44,8 +44,14 @@ COPY install/ubuntu_llvm.sh /work/ RUN /work/ubuntu_llvm.sh COPY install/ubuntu_caffe.sh /work/ RUN /work/ubuntu_caffe.sh +COPY install/ubuntu_onnx.sh /work/ +RUN /work/ubuntu_onnx.sh COPY install/ubuntu_adduser.sh /work/ Review comment: will update This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
