[incubator-mxnet] branch master updated (7dde0eb -> d60f37b)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 7dde0eb [MXNET-1234] Fix shape inference problems in Activation backward (#13409) add d60f37b Docs & website sphinx errors squished (#13488) No new revisions were added by this update. Summary of changes: docs/api/scala/index.md | 18 +-- docs/api/scala/kvstore.md| 98 +++--- docs/api/scala/ndarray.md| 186 +-- docs/api/scala/symbol.md | 66 +- docs/gluon/index.md | 10 +- docs/install/ubuntu_setup.md | 12 -- docs/tutorials/r/fiveMinutesNeuralNetwork.md | 4 +- python/mxnet/gluon/parameter.py | 2 +- 8 files changed, 192 insertions(+), 204 deletions(-)
[incubator-mxnet] branch master updated: fix toctree Sphinx errors (#13489)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new f2dcd7c fix toctree Sphinx errors (#13489) f2dcd7c is described below commit f2dcd7c7b8676b55d912997fc3f9c62c55915307 Author: Aaron Markham AuthorDate: Mon Dec 3 17:27:41 2018 -0800 fix toctree Sphinx errors (#13489) * fix toctree errors * nudging file for CI --- docs/api/index.md | 2 ++ docs/tutorials/index.md | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/api/index.md b/docs/api/index.md index eff6807..9e7a58f 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -1,11 +1,13 @@ # MXNet APIs + ```eval_rst .. toctree:: :maxdepth: 1 c++/index.md clojure/index.md + java/index.md julia/index.md perl/index.md python/index.md diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 52e2be8..7d102bb 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -3,12 +3,13 @@ ```eval_rst .. toctree:: :hidden: - + basic/index.md c++/index.md control_flow/index.md embedded/index.md gluon/index.md + java/index.md nlp/index.md onnx/index.md python/index.md
[incubator-mxnet] branch master updated (3c4a97d -> 3a50ae0)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 3c4a97d Fix github status context if running inside a folder (#13350) add 3a50ae0 Fix sphinx docstring warnings. (#13315) No new revisions were added by this update. Summary of changes: python/mxnet/contrib/io.py| 4 ++-- python/mxnet/contrib/svrg_optimization/svrg_module.py | 16 python/mxnet/contrib/text/embedding.py| 6 -- python/mxnet/gluon/data/vision/datasets.py| 19 ++- python/mxnet/image/image.py | 4 ++-- python/mxnet/optimizer/optimizer.py | 3 ++- python/mxnet/rnn/rnn_cell.py | 3 ++- python/mxnet/symbol/contrib.py| 18 +- python/mxnet/torch.py | 2 +- src/operator/contrib/adaptive_avg_pooling.cc | 4 ++-- 10 files changed, 42 insertions(+), 37 deletions(-)
[incubator-mxnet] branch master updated: Fix descriptions in scaladocs for macro ndarray/sybmol APIs (#13210)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new dfeb6b0 Fix descriptions in scaladocs for macro ndarray/sybmol APIs
(#13210)
dfeb6b0 is described below
commit dfeb6b017445b4a6b37a67abc8d8e23e3b8f4a0a
Author: Zach Kimberg
AuthorDate: Fri Nov 16 16:49:04 2018 -0800
Fix descriptions in scaladocs for macro ndarray/sybmol APIs (#13210)
---
.../src/main/scala/org/apache/mxnet/APIDocGenerator.scala | 15 ---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git
a/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
b/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
index 7c1edb5..ce12dc7 100644
--- a/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
+++ b/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
@@ -114,11 +114,20 @@ private[mxnet] object APIDocGenerator extends
GeneratorBase {
}
def generateAPIDocFromBackend(func: Func, withParam: Boolean = true): String
= {
-val desc = func.desc.split("\n")
- .mkString(" * \n", "\n * ", " * \n")
+def fixDesc(desc: String): String = {
+ var curDesc = desc
+ var prevDesc = ""
+ while ( curDesc != prevDesc ) {
+prevDesc = curDesc
+curDesc = curDesc.replace("[[", "`[ [").replace("]]", "] ]")
+ }
+ curDesc
+}
+val desc = fixDesc(func.desc).split("\n")
+ .mkString(" *\n * {{{\n *\n * ", "\n * ", "\n * }}}\n * ")
val params = func.listOfArgs.map { absClassArg =>
- s" * @param ${absClassArg.safeArgName}\t\t${absClassArg.argDesc}"
+ s" * @param
${absClassArg.safeArgName}\t\t${fixDesc(absClassArg.argDesc)}"
}
val returnType = s" * @return ${func.returnType}"
[incubator-mxnet] branch master updated (46e870b -> 1aa6a38)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 46e870b fix the flag (#13293) add 1aa6a38 Made fixes to sparse.py and sparse.md (#13305) No new revisions were added by this update. Summary of changes: docs/api/python/ndarray/sparse.md | 1 + python/mxnet/ndarray/sparse.py| 70 +++ 2 files changed, 36 insertions(+), 35 deletions(-)
[incubator-mxnet] branch master updated: Addressed "dumplicate object reference" issues (#13214)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 6ae5b65 Addressed "dumplicate object reference" issues (#13214) 6ae5b65 is described below commit 6ae5b657eba67c565c08012889777fd7ca4552c4 Author: vdantu <36211508+vda...@users.noreply.github.com> AuthorDate: Fri Nov 16 14:13:05 2018 -0800 Addressed "dumplicate object reference" issues (#13214) --- docs/api/python/symbol/random.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/api/python/symbol/random.md b/docs/api/python/symbol/random.md index 22c686f..b93f641 100644 --- a/docs/api/python/symbol/random.md +++ b/docs/api/python/symbol/random.md @@ -48,6 +48,7 @@ In the rest of this document, we list routines provided by the `symbol.random` p .. automodule:: mxnet.symbol.random :members: +:noindex: .. automodule:: mxnet.random :members:
[incubator-mxnet] branch master updated: Visualization doc fix. Added notes for shortform (#13291)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 5af975e Visualization doc fix. Added notes for shortform (#13291)
5af975e is described below
commit 5af975e8b1d5f368f5c9a039ec6d2184c583eb99
Author: vdantu <36211508+vda...@users.noreply.github.com>
AuthorDate: Fri Nov 16 13:35:10 2018 -0800
Visualization doc fix. Added notes for shortform (#13291)
---
python/mxnet/visualization.py | 25 +
1 file changed, 25 insertions(+)
diff --git a/python/mxnet/visualization.py b/python/mxnet/visualization.py
index 9297ede..1ebdcb5 100644
--- a/python/mxnet/visualization.py
+++ b/python/mxnet/visualization.py
@@ -57,9 +57,22 @@ def print_summary(symbol, shape=None, line_length=120,
positions=[.44, .64, .74,
Rotal length of printed lines
positions: list
Relative or absolute positions of log elements in each line.
+
Returns
--
None
+
+Notes
+-
+If ``mxnet`` is imported, the visualization module can be used in its
short-form.
+For example, if we ``import mxnet`` as follows::
+
+import mxnet
+
+this method in visualization module can be used in its short-form as::
+
+mxnet.viz.print_summary(...)
+
"""
if not isinstance(symbol, Symbol):
raise TypeError("symbol must be Symbol")
@@ -238,6 +251,18 @@ def plot_network(symbol, title="plot", save_format='pdf',
shape=None, node_attrs
>>> digraph = mx.viz.plot_network(net, shape={'data':(100,200)},
... node_attrs={"fixedsize":"false"})
>>> digraph.view()
+
+Notes
+-
+If ``mxnet`` is imported, the visualization module can be used in its
short-form.
+For example, if we ``import mxnet`` as follows::
+
+import mxnet
+
+this method in visualization module can be used in its short-form as::
+
+mxnet.viz.plot_network(...)
+
"""
# todo add shape support
try:
[incubator-mxnet] branch master updated (20a23ef -> 21fc3af)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 20a23ef Sphinx errors in Gluon (#13275) add 21fc3af Fix Sphinx python docstring formatting error. (#13194) No new revisions were added by this update. Summary of changes: src/operator/nn/batch_norm.cc | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-)
[incubator-mxnet] branch master updated: stop gap fix to let website builds through; scaladoc fix pending (#13298)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new d73e4df stop gap fix to let website builds through; scaladoc fix
pending (#13298)
d73e4df is described below
commit d73e4df122302bd47790ff42810f6e71989420b1
Author: Aaron Markham
AuthorDate: Fri Nov 16 10:30:45 2018 -0800
stop gap fix to let website builds through; scaladoc fix pending (#13298)
---
docs/mxdoc.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/mxdoc.py b/docs/mxdoc.py
index 8b26c89..5e86c1c 100644
--- a/docs/mxdoc.py
+++ b/docs/mxdoc.py
@@ -115,7 +115,7 @@ def build_scala_docs(app):
'`find native -name "*.jar" | grep "target/lib/" | tr "\\n" ":" `',
'`find macros -name "*-SNAPSHOT.jar" | tr "\\n" ":" `'
])
-_run_cmd('cd {}; scaladoc `{}` -classpath {} -feature -deprecation'
+_run_cmd('cd {}; scaladoc `{}` -classpath {} -feature -deprecation; exit 0'
.format(scala_path, scala_doc_sources, scala_doc_classpath))
dest_path = app.builder.outdir + '/api/scala/docs'
_run_cmd('rm -rf ' + dest_path)
[incubator-mxnet] branch master updated: [MXNET-1203] Tutorial infogan (#13144)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 97fdfd9 [MXNET-1203] Tutorial infogan (#13144)
97fdfd9 is described below
commit 97fdfd947e841b2556c8a8a19bdeed8f4e7bd407
Author: Nathalie Rauschmayr
AuthorDate: Thu Nov 15 15:08:19 2018 -0800
[MXNET-1203] Tutorial infogan (#13144)
* Adding info_gan example
* adjust paths of filenames
* Update index.md
* Update index.md
* Update index.md
* Update info_gan.md
Added an image
* Update info_gan.md
Applied some fixes
* Update info_gan.md
Applied some fixes
* Update info_gan.md
Applied some fixes
* Update info_gan.md
* Updated index.md file
* Updated index.md file
* change links
* Fixed typo
* Delete Untitled.ipynb
* Adding Vishaals comments
* Adding Anirudh's comments
* Fixed some bugs
* Adding Anirudh's comments
* some minor fixes
---
docs/tutorials/gluon/info_gan.md | 437 ++
docs/tutorials/index.md | 1 +
tests/tutorials/test_tutorials.py | 3 +
3 files changed, 441 insertions(+)
diff --git a/docs/tutorials/gluon/info_gan.md b/docs/tutorials/gluon/info_gan.md
new file mode 100644
index 000..c8f07c6
--- /dev/null
+++ b/docs/tutorials/gluon/info_gan.md
@@ -0,0 +1,437 @@
+
+# Image similarity search with InfoGAN
+
+This notebook shows how to implement an InfoGAN based on Gluon. InfoGAN is an
extension of GANs, where the generator input is split in 2 parts: random noise
and a latent code (see [InfoGAN Paper](https://arxiv.org/pdf/1606.03657.pdf)).
+The codes are made meaningful by maximizing the mutual information between
code and generator output. InfoGAN learns a disentangled representation in a
completely unsupervised manner. It can be used for many applications such as
image similarity search. This notebook uses the DCGAN example from the
[Straight Dope
Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html)
and extends it to create an InfoGAN.
+
+
+```python
+from __future__ import print_function
+from datetime import datetime
+import logging
+import multiprocessing
+import os
+import sys
+import tarfile
+import time
+
+import numpy as np
+from matplotlib import pyplot as plt
+from mxboard import SummaryWriter
+import mxnet as mx
+from mxnet import gluon
+from mxnet import ndarray as nd
+from mxnet.gluon import nn, utils
+from mxnet import autograd
+
+```
+
+The latent code vector can contain several variables, which can be categorical
and/or continuous. We set `n_continuous` to 2 and `n_categories` to 10.
+
+
+```python
+batch_size = 64
+z_dim= 100
+n_continuous = 2
+n_categories = 10
+ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()
+```
+
+Some functions to load and normalize images.
+
+
+```python
+lfw_url = 'http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz'
+data_path = 'lfw_dataset'
+if not os.path.exists(data_path):
+os.makedirs(data_path)
+data_file = utils.download(lfw_url)
+with tarfile.open(data_file) as tar:
+tar.extractall(path=data_path)
+
+```
+
+
+```python
+def transform(data, width=64, height=64):
+data = mx.image.imresize(data, width, height)
+data = nd.transpose(data, (2,0,1))
+data = data.astype(np.float32)/127.5 - 1
+if data.shape[0] == 1:
+data = nd.tile(data, (3, 1, 1))
+return data.reshape((1,) + data.shape)
+```
+
+
+```python
+def get_files(data_dir):
+images= []
+filenames = []
+for path, _, fnames in os.walk(data_dir):
+for fname in fnames:
+if not fname.endswith('.jpg'):
+continue
+img = os.path.join(path, fname)
+img_arr = mx.image.imread(img)
+img_arr = transform(img_arr)
+images.append(img_arr)
+filenames.append(path + "/" + fname)
+return images, filenames
+```
+
+Load the dataset `lfw_dataset` which contains images of celebrities.
+
+
+```python
+data_dir = 'lfw_dataset'
+images, filenames = get_files(data_dir)
+split = int(len(images)*0.8)
+test_images = images[split:]
+test_filenames = filenames[split:]
+train_images = images[:split]
+train_filenames = filenames[:split]
+
+train_data = gluon.data.ArrayDataset(nd.concatenate(train_images))
+train_dataloader = gluon.data.DataLoader(train_data, batch_size=batch_size,
shuffle=True, last_batch='rollover', num_workers=multiprocessing.cpu_count()-1)
+```
+
+## Generator
+Define the Generator model. Architecture is taken from the DCGAN
implementation in [Straight Dope
Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/
[incubator-mxnet] branch master updated: [Example] Gradcam consolidation in tutorial (#13255)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 8ac7fb9 [Example] Gradcam consolidation in tutorial (#13255)
8ac7fb9 is described below
commit 8ac7fb930fdfa6ef3ac61be7569a17eb95f1ad4c
Author: Ankit Khedia <36249596+ankkhe...@users.noreply.github.com>
AuthorDate: Thu Nov 15 15:05:36 2018 -0800
[Example] Gradcam consolidation in tutorial (#13255)
* fixing gradcam
* changed loading parameters code
* fixing type conversions issue with previous versions of matplotlib
* gradcam consolidation
* creating directory structures in utils
* changing location
* empty commit
---
docs/conf.py | 2 +-
.../vision}/cnn_visualization/gradcam.py | 0
docs/tutorials/vision/cnn_visualization.md | 3 +-
example/cnn_visualization/README.md| 17
example/cnn_visualization/gradcam_demo.py | 110 -
example/cnn_visualization/vgg.py | 90 -
6 files changed, 3 insertions(+), 219 deletions(-)
diff --git a/docs/conf.py b/docs/conf.py
index 656a1da..af23521 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -107,7 +107,7 @@ master_doc = 'index'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
-exclude_patterns = ['3rdparty', 'build_version_doc', 'virtualenv',
'api/python/model.md', 'README.md']
+exclude_patterns = ['3rdparty', 'build_version_doc', 'virtualenv',
'api/python/model.md', 'README.md', 'tutorial_utils']
# The reST default role (used for this markup: `text`) to use for all
documents.
#default_role = None
diff --git a/example/cnn_visualization/gradcam.py
b/docs/tutorial_utils/vision/cnn_visualization/gradcam.py
similarity index 100%
rename from example/cnn_visualization/gradcam.py
rename to docs/tutorial_utils/vision/cnn_visualization/gradcam.py
diff --git a/docs/tutorials/vision/cnn_visualization.md
b/docs/tutorials/vision/cnn_visualization.md
index a350fff..fd6a464 100644
--- a/docs/tutorials/vision/cnn_visualization.md
+++ b/docs/tutorials/vision/cnn_visualization.md
@@ -22,7 +22,7 @@ from matplotlib import pyplot as plt
import numpy as np
gradcam_file = "gradcam.py"
-base_url =
"https://raw.githubusercontent.com/indhub/mxnet/cnnviz/example/cnn_visualization/{}?raw=true;
+base_url =
"https://github.com/apache/incubator-mxnet/tree/master/docs/tutorial_utils/vision/cnn_visualization/{}?raw=true;
mx.test_utils.download(base_url.format(gradcam_file), fname=gradcam_file)
import gradcam
```
@@ -182,6 +182,7 @@ Next, we'll write a method to get an image, preprocess it,
predict category and
2. **Guided Grad-CAM:** Guided Grad-CAM shows which exact pixels contributed
the most to the CNN's decision.
3. **Saliency map:** Saliency map is a monochrome image showing which pixels
contributed the most to the CNN's decision. Sometimes, it is easier to see the
areas in the image that most influence the output in a monochrome image than in
a color image.
+
```python
def visualize(net, img_path, conv_layer_name):
orig_img = mx.img.imread(img_path)
diff --git a/example/cnn_visualization/README.md
b/example/cnn_visualization/README.md
deleted file mode 100644
index 10b9149..000
--- a/example/cnn_visualization/README.md
+++ /dev/null
@@ -1,17 +0,0 @@
-# Visualzing CNN decisions
-
-This folder contains an MXNet Gluon implementation of
[Grad-CAM](https://arxiv.org/abs/1610.02391) that helps visualize CNN decisions.
-
-A tutorial on how to use this from Jupyter notebook is available
[here](https://mxnet.incubator.apache.org/tutorials/vision/cnn_visualization.html).
-
-You can also do the visualization from terminal:
-```
-$ python gradcam_demo.py hummingbird.jpg
-Predicted category : hummingbird (94)
-Original Image : hummingbird_orig.jpg
-Grad-CAM: hummingbird_gradcam.jpg
-Guided Grad-CAM : hummingbird_guided_gradcam.jpg
-Saliency Map: hummingbird_saliency.jpg
-```
-
-
diff --git a/example/cnn_visualization/gradcam_demo.py
b/example/cnn_visualization/gradcam_demo.py
deleted file mode 100644
index d9ca5dd..000
--- a/example/cnn_visualization/gradcam_demo.py
+++ /dev/null
@@ -1,110 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"
[incubator-mxnet] branch master updated: [Example] Fixing Gradcam implementation (#13196)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new e655f62 [Example] Fixing Gradcam implementation (#13196)
e655f62 is described below
commit e655f62bcccdf55fbc62b96cd6b12e7fbe68aaba
Author: Ankit Khedia <36249596+ankkhe...@users.noreply.github.com>
AuthorDate: Tue Nov 13 15:38:06 2018 -0800
[Example] Fixing Gradcam implementation (#13196)
* fixing gradcam
* changed loading parameters code
* fixing type conversions issue with previous versions of matplotlib
---
docs/tutorials/vision/cnn_visualization.md | 3 ++-
example/cnn_visualization/gradcam.py | 4 ++--
example/cnn_visualization/vgg.py | 16 +++-
3 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/docs/tutorials/vision/cnn_visualization.md
b/docs/tutorials/vision/cnn_visualization.md
index 940c261..a350fff 100644
--- a/docs/tutorials/vision/cnn_visualization.md
+++ b/docs/tutorials/vision/cnn_visualization.md
@@ -151,7 +151,8 @@ def show_images(pred_str, images):
for i in range(num_images):
fig.add_subplot(rows, cols, i+1)
plt.xlabel(titles[i])
-plt.imshow(images[i], cmap='gray' if i==num_images-1 else None)
+img = images[i].astype(np.uint8)
+plt.imshow(img, cmap='gray' if i==num_images-1 else None)
plt.show()
```
diff --git a/example/cnn_visualization/gradcam.py
b/example/cnn_visualization/gradcam.py
index a8708f7..54cb65e 100644
--- a/example/cnn_visualization/gradcam.py
+++ b/example/cnn_visualization/gradcam.py
@@ -249,8 +249,8 @@ def visualize(net, preprocessed_img, orig_img,
conv_layer_name):
imggrad = get_image_grad(net, preprocessed_img)
conv_out, conv_out_grad = get_conv_out_grad(net, preprocessed_img,
conv_layer_name=conv_layer_name)
-cam = get_cam(imggrad, conv_out)
-
+cam = get_cam(conv_out_grad, conv_out)
+cam = cv2.resize(cam, (imggrad.shape[1], imggrad.shape[2]))
ggcam = get_guided_grad_cam(cam, imggrad)
img_ggcam = grad_to_image(ggcam)
diff --git a/example/cnn_visualization/vgg.py b/example/cnn_visualization/vgg.py
index b6215a3..a8a0ef6 100644
--- a/example/cnn_visualization/vgg.py
+++ b/example/cnn_visualization/vgg.py
@@ -72,11 +72,17 @@ def get_vgg(num_layers, pretrained=False, ctx=mx.cpu(),
root=os.path.join('~', '.mxnet', 'models'), **kwargs):
layers, filters = vgg_spec[num_layers]
net = VGG(layers, filters, **kwargs)
-if pretrained:
-from mxnet.gluon.model_zoo.model_store import get_model_file
-batch_norm_suffix = '_bn' if kwargs.get('batch_norm') else ''
-net.load_params(get_model_file('vgg%d%s'%(num_layers,
batch_norm_suffix),
- root=root), ctx=ctx)
+net.initialize(ctx=ctx)
+
+# Get the pretrained model
+vgg = mx.gluon.model_zoo.vision.get_vgg(num_layers, pretrained=True,
ctx=ctx)
+
+# Set the parameters in the new network
+params = vgg.collect_params()
+for key in params:
+param = params[key]
+net.collect_params()[net.prefix+key.replace(vgg.prefix,
'')].set_data(param.data())
+
return net
def vgg16(**kwargs):
[incubator-mxnet] branch master updated: Add Java API docs generation (#13071)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new ea6ee0d Add Java API docs generation (#13071)
ea6ee0d is described below
commit ea6ee0d217dcdac29fe0a22bdce019168a76a060
Author: Aaron Markham
AuthorDate: Tue Nov 13 12:17:44 2018 -0800
Add Java API docs generation (#13071)
* add Java API docs generation; split out from Scala API docs
* bumping file for ci
* make scala docs build compatible for 2.11.x and 2.12.x scala
fix typo
* fix exit bug
---
docs/mxdoc.py | 31 ---
docs/settings.ini | 11 +++
2 files changed, 35 insertions(+), 7 deletions(-)
diff --git a/docs/mxdoc.py b/docs/mxdoc.py
index 7092b9e..8570cae 100644
--- a/docs/mxdoc.py
+++ b/docs/mxdoc.py
@@ -40,11 +40,12 @@ if _DOC_SET not in parser.sections():
for section in [ _DOC_SET ]:
print("Document sets to generate:")
-for candidate in [ 'scala_docs', 'clojure_docs', 'doxygen_docs', 'r_docs'
]:
+for candidate in [ 'scala_docs', 'java_docs', 'clojure_docs',
'doxygen_docs', 'r_docs' ]:
print('%-12s : %s' % (candidate, parser.get(section, candidate)))
_MXNET_DOCS_BUILD_MXNET = parser.getboolean('mxnet', 'build_mxnet')
_SCALA_DOCS = parser.getboolean(_DOC_SET, 'scala_docs')
+_JAVA_DOCS = parser.getboolean(_DOC_SET, 'java_docs')
_CLOJURE_DOCS = parser.getboolean(_DOC_SET, 'clojure_docs')
_DOXYGEN_DOCS = parser.getboolean(_DOC_SET, 'doxygen_docs')
_R_DOCS = parser.getboolean(_DOC_SET, 'r_docs')
@@ -58,7 +59,8 @@ _CODE_MARK = re.compile('^([ ]*)```([\w]*)')
# language names and the according file extensions and comment symbol
_LANGS = {'python' : ('py', '#'),
'r' : ('R','#'),
- 'scala' : ('scala', '#'),
+ 'scala' : ('scala', '//'),
+ 'java' : ('java', '//'),
'julia' : ('jl', '#'),
'perl' : ('pl', '#'),
'cpp' : ('cc', '//'),
@@ -101,7 +103,7 @@ def build_r_docs(app):
_run_cmd('mkdir -p ' + dest_path + '; mv ' + pdf_path + ' ' + dest_path)
def build_scala(app):
-"""build scala for scala docs and clojure docs to use"""
+"""build scala for scala docs, java docs, and clojure docs to use"""
_run_cmd("cd %s/.. && make scalapkg" % app.builder.srcdir)
_run_cmd("cd %s/.. && make scalainstall" % app.builder.srcdir)
@@ -109,13 +111,26 @@ def build_scala_docs(app):
"""build scala doc and then move the outdir"""
scala_path = app.builder.srcdir + '/../scala-package'
# scaldoc fails on some apis, so exit 0 to pass the check
-_run_cmd('cd ' + scala_path + '; scaladoc `find . -type f -name "*.scala"
| egrep \"\/core|\/infer\" | egrep -v \"Suite\"`; exit 0')
+_run_cmd('cd ' + scala_path + '; scaladoc `find . -type f -name "*.scala"
| egrep \"\/core|\/infer\" | egrep -v \"Suite|javaapi\"`; exit 0')
dest_path = app.builder.outdir + '/api/scala/docs'
_run_cmd('rm -rf ' + dest_path)
_run_cmd('mkdir -p ' + dest_path)
+# 'index' and 'package.html' do not exist in later versions of scala;
delete these after upgrading scala>2.12.x
scaladocs = ['index', 'index.html', 'org', 'lib', 'index.js',
'package.html']
for doc_file in scaladocs:
-_run_cmd('cd ' + scala_path + ' && mv -f ' + doc_file + ' ' +
dest_path)
+_run_cmd('cd ' + scala_path + ' && mv -f ' + doc_file + ' ' +
dest_path + '; exit 0')
+
+def build_java_docs(app):
+"""build java docs and then move the outdir"""
+java_path = app.builder.srcdir +
'/../scala-package/core/src/main/scala/org/apache/mxnet/'
+# scaldoc fails on some apis, so exit 0 to pass the check
+_run_cmd('cd ' + java_path + '; scaladoc `find . -type f -name "*.scala" |
egrep \"\/javaapi\" | egrep -v \"Suite\"`; exit 0')
+dest_path = app.builder.outdir + '/api/java/docs'
+_run_cmd('rm -rf ' + dest_path)
+_run_cmd('mkdir -p ' + dest_path)
+javadocs = ['index', 'index.html', 'org', 'lib', 'index.js',
'package.html']
+for doc_file in javadocs:
+_run_cmd('cd ' + java_path + ' && mv -f ' + doc_file + ' ' + dest_path
+ '; exit 0')
def build_clojure_docs(app):
"""build clojure doc and then move the outdir"""
@@ -125,7 +140,7 @@ def build_clojure_docs(app):
_run_cmd('rm -rf ' + dest_path)
_run_cmd('mkdir -p ' + dest_path)
clojure_doc_path = app.builder.srcdir +
'/../contrib/clojure-package/target/doc'
-_run_cmd('cd ' + clojure_doc_path + ' && cp -r * ' + dest_path)
+
[incubator-mxnet] branch master updated: Updates to several examples (#13068)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 012288f Updates to several examples (#13068) 012288f is described below commit 012288f0325d58582d1b12499e9daccdad696080 Author: Thomas Delteil AuthorDate: Thu Nov 8 09:53:33 2018 -0800 Updates to several examples (#13068) * Minor updates to several examples * fix typo * update following review --- example/reinforcement-learning/ddpg/README.md | 2 + example/reinforcement-learning/dqn/setup.sh| 6 +- example/restricted-boltzmann-machine/README.md | 52 example/rnn-time-major/bucket_io.py| 264 - example/rnn-time-major/get_sherlockholmes_data.sh | 43 example/rnn-time-major/readme.md | 24 -- example/rnn-time-major/rnn_cell_demo.py| 189 --- example/rnn/README.md | 5 + example/rnn/bucketing/README.md| 13 +- example/rnn/large_word_lm/data.py | 2 +- example/rnn/large_word_lm/readme.md| 66 -- example/rnn/word_lm/README.md | 2 +- example/sparse/linear_classification/data.py | 5 +- example/speech_recognition/README.md | 4 +- example/speech_recognition/label_util.py | 2 +- example/speech_recognition/log_util.py | 74 +++--- example/speech_recognition/main.py | 6 +- example/speech_recognition/singleton.py| 32 ++- example/speech_recognition/stt_datagenerator.py| 6 +- example/speech_recognition/stt_io_iter.py | 6 +- example/speech_recognition/stt_metric.py | 2 +- example/speech_recognition/stt_utils.py| 6 +- example/speech_recognition/train.py| 2 +- example/ssd/README.md | 8 + example/stochastic-depth/sd_cifar10.py | 31 ++- example/stochastic-depth/sd_mnist.py | 6 +- example/svm_mnist/svm_mnist.py | 110 + example/svrg_module/README.md | 6 +- .../svrg_module/benchmarks/svrg_benchmark.ipynb| 111 - .../svrg_module/linear_regression/data_reader.py | 24 +- example/svrg_module/linear_regression/train.py | 4 +- 31 files changed, 326 insertions(+), 787 deletions(-) diff --git a/example/reinforcement-learning/ddpg/README.md b/example/reinforcement-learning/ddpg/README.md index 37f42a8..2e299dd 100644 --- a/example/reinforcement-learning/ddpg/README.md +++ b/example/reinforcement-learning/ddpg/README.md @@ -1,6 +1,8 @@ # mx-DDPG MXNet Implementation of DDPG +## /!\ This example depends on RLLAB which is deprecated /!\ + # Introduction This is the MXNet implementation of [DDPG](https://arxiv.org/abs/1509.02971). It is tested in the rllab cart pole environment against rllab's native implementation and achieves comparably similar results. You can substitute with this anywhere you use rllab's DDPG with minor modifications. diff --git a/example/reinforcement-learning/dqn/setup.sh b/example/reinforcement-learning/dqn/setup.sh index 3069fef..012ff8f 100755 --- a/example/reinforcement-learning/dqn/setup.sh +++ b/example/reinforcement-learning/dqn/setup.sh @@ -26,11 +26,11 @@ pip install pygame # Install arcade learning environment if [[ "$OSTYPE" == "linux-gnu" ]]; then -sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake +sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake ninja-build elif [[ "$OSTYPE" == "darwin"* ]]; then brew install sdl sdl_image sdl_mixer sdl_ttf portmidi fi -git clone g...@github.com:mgbellemare/Arcade-Learning-Environment.git || true +git clone https://github.com/mgbellemare/Arcade-Learning-Environment || true pushd . cd Arcade-Learning-Environment mkdir -p build @@ -43,6 +43,6 @@ popd cp Arcade-Learning-Environment/ale.cfg . # Copy roms -git clone g...@github.com:npow/atari.git || true +git clone https://github.com/npow/atari || true cp -R atari/roms . diff --git a/example/restricted-boltzmann-machine/README.md b/example/restricted-boltzmann-machine/README.md index 129120b..a8769a5 100644 --- a/example/restricted-boltzmann-machine/README.md +++ b/example/restricted-boltzmann-machine/README.md @@ -8,6 +8,58 @@ Here are some samples generated by the RBM with the default hyperparameters. The +Usage: + +``` +python binary_rbm_gluon.py --help +usage: binary_rbm_gluon.py [-h] [--num-hidden NUM_HIDDEN] [--k K] + [--batch-size BATCH_SIZE] [--num-epoch NUM_EPOCH] + [--learning-rate LEARNING_RATE] +
[incubator-mxnet] branch master updated: Updated capsnet example (#12934)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 68bc9b7 Updated capsnet example (#12934) 68bc9b7 is described below commit 68bc9b7f444e76e42c02adfa97ec12149ba0d996 Author: Thomas Delteil AuthorDate: Thu Nov 8 08:38:42 2018 -0800 Updated capsnet example (#12934) * Updated capsnet * trigger CI * Update README.md --- example/capsnet/README.md | 132 example/capsnet/capsulenet.py | 695 +- 2 files changed, 413 insertions(+), 414 deletions(-) diff --git a/example/capsnet/README.md b/example/capsnet/README.md index 49a6dd1..500c7df 100644 --- a/example/capsnet/README.md +++ b/example/capsnet/README.md @@ -1,66 +1,66 @@ -**CapsNet-MXNet** -= - -This example is MXNet implementation of [CapsNet](https://arxiv.org/abs/1710.09829): -Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017 -- The current `best test error is 0.29%` and `average test error is 0.303%` -- The `average test error on paper is 0.25%` - -Log files for the error rate are uploaded in [repository](https://github.com/samsungsds-rnd/capsnet.mxnet). -* * * -## **Usage** -Install scipy with pip -``` -pip install scipy -``` -Install tensorboard with pip -``` -pip install tensorboard -``` - -On Single gpu -``` -python capsulenet.py --devices gpu0 -``` -On Multi gpus -``` -python capsulenet.py --devices gpu0,gpu1 -``` -Full arguments -``` -python capsulenet.py --batch_size 100 --devices gpu0,gpu1 --num_epoch 100 --lr 0.001 --num_routing 3 --model_prefix capsnet -``` - -* * * -## **Prerequisities** - -MXNet version above (0.11.0) -scipy version above (0.19.0) - -*** -## **Results** -Train time takes about 36 seconds for each epoch (batch_size=100, 2 gtx 1080 gpus) - -CapsNet classification test error on MNIST - -``` -python capsulenet.py --devices gpu0,gpu1 --lr 0.0005 --decay 0.99 --model_prefix lr_0_0005_decay_0_99 --batch_size 100 --num_routing 3 --num_epoch 200 -``` - - - -| Trial | Epoch | train err(%) | test err(%) | train loss | test loss | -| :---: | :---: | :---: | :---: | :---: | :---: | -| 1 | 120 | 0.06 | 0.31 | 0.0056 | 0.0064 | -| 2 | 167 | 0.03 | 0.29 | 0.0048 | 0.0058 | -| 3 | 182 | 0.04 | 0.31 | 0.0046 | 0.0058 | -| average | - | 0.043 | 0.303 | 0.005 | 0.006 | - -We achieved `the best test error rate=0.29%` and `average test error=0.303%`. It is the best accuracy and fastest training time result among other implementations(Keras, Tensorflow at 2017-11-23). -The result on paper is `0.25% (average test error rate)`. - -| Implementation| test err(%) | ※train time/epoch | GPU Used| -| :---: | :---: | :---: |:---: | -| MXNet | 0.29 | 36 sec | 2 GTX 1080 | -| tensorflow | 0.49 | ※ 10 min | Unknown(4GB Memory) | -| Keras | 0.30 | 55 sec | 2 GTX 1080 Ti | +**CapsNet-MXNet** += + +This example is MXNet implementation of [CapsNet](https://arxiv.org/abs/1710.09829): +Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017 +- The current `best test error is 0.29%` and `average test error is 0.303%` +- The `average test error on paper is 0.25%` + +Log files for the error rate are uploaded in [repository](https://github.com/samsungsds-rnd/capsnet.mxnet). +* * * +## **Usage** +Install scipy with pip +``` +pip install scipy +``` +Install tensorboard and mxboard with pip +``` +pip install mxboard tensorflow +``` + +On Single gpu +``` +python capsulenet.py --devices gpu0 +``` +On Multi gpus +``` +python capsulenet.py --devices gpu0,gpu1 +``` +Full arguments +``` +python capsulenet.py --batch_size 100 --devices gpu0,gpu1 --num_epoch 100 --lr 0.001 --num_routing 3 --model_prefix capsnet +``` + +* * * +## **Prerequisities** + +MXNet version above (1.2.0) +scipy version above (0.19.0) + +*** +## **Results** +Train time takes about 36 seconds for each epoch (batch_size=100, 2 gtx 1080 gpus) + +CapsNet classification test error on MNIST: + +``` +python capsulenet.py --devices gpu0,gpu1 --lr 0.0005 --decay 0.99 --model_prefix lr_0_0005_decay_0_99 --batch_size 100 --num_routing 3 --num_epoch 200 +``` + + + +| Trial | Epoch | train err(%) | test err(%) | train loss | test loss | +| :---: | :---: | :---: | :---: | :---: | :---: | +| 1 | 120 | 0.06 | 0.31 | 0.0056 | 0.0064 | +| 2 | 167 | 0.03 | 0.29 | 0.0048 | 0.0058 | +| 3 | 182 | 0.04 | 0.31 | 0.0046 | 0.0058 | +| average | - | 0.043 | 0.303 | 0.005 | 0.006 | + +We achieved `the best test error rate=0.29%` and `average test error=0.303%`. It is the best accuracy and fastest training time result among other implementations(Keras, Tensorflow at 2017-11-23). +The result on paper
[incubator-mxnet] branch master updated: Update Gluon example folder (#12951)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c169b14 Update Gluon example folder (#12951)
c169b14 is described below
commit c169b14b35cb8eb25df166d2e5f5ddc1c4fa5d5f
Author: Thomas Delteil
AuthorDate: Tue Nov 6 11:01:43 2018 -0800
Update Gluon example folder (#12951)
* Reorganized the Gluon folder in example
* trigger CI
* update reference
* fix out of place accumulation
---
docs/tutorials/unsupervised_learning/gan.md| 2 +-
example/gluon/{ => actor_critic}/actor_critic.py | 0
example/gluon/{DCGAN => dc_gan}/README.md | 0
example/gluon/{DCGAN => dc_gan}/__init__.py| 0
example/gluon/{DCGAN => dc_gan}/dcgan.py | 0
example/gluon/{DCGAN => dc_gan}/inception_score.py | 0
.../kaggle_k_fold_cross_validation.py | 0
example/gluon/learning_rate_manipulation.py| 63 --
example/gluon/{ => lstm_crf}/lstm_crf.py | 10 ++--
example/gluon/{ => mnist}/mnist.py | 0
example/gluon/sn_gan/data.py | 2 +-
example/gluon/sn_gan/model.py | 2 +-
example/gluon/sn_gan/train.py | 2 +-
example/gluon/sn_gan/utils.py | 2 +-
.../{ => super_resolution}/super_resolution.py | 0
example/notebooks/README.md| 4 --
16 files changed, 12 insertions(+), 75 deletions(-)
diff --git a/docs/tutorials/unsupervised_learning/gan.md
b/docs/tutorials/unsupervised_learning/gan.md
index 1556bf6..f436a15 100644
--- a/docs/tutorials/unsupervised_learning/gan.md
+++ b/docs/tutorials/unsupervised_learning/gan.md
@@ -394,7 +394,7 @@ As a result, we have created two neural nets: a Generator,
which is able to crea
Along the way, we have learned how to do the image manipulation and
visualization that is associated with the training of deep neural nets. We have
also learned how to use MXNet's Module APIs to perform advanced model training
functionality to fit the model.
## Acknowledgements
-This tutorial is based on [MXNet DCGAN
codebase](https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dcgan.py),
+This tutorial is based on [MXNet DCGAN
codebase](https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py),
[The original paper on GANs](https://arxiv.org/abs/1406.2661), as well as
[this paper on deep convolutional GANs](https://arxiv.org/abs/1511.06434).
\ No newline at end of file
diff --git a/example/gluon/actor_critic.py
b/example/gluon/actor_critic/actor_critic.py
similarity index 100%
rename from example/gluon/actor_critic.py
rename to example/gluon/actor_critic/actor_critic.py
diff --git a/example/gluon/DCGAN/README.md b/example/gluon/dc_gan/README.md
similarity index 100%
rename from example/gluon/DCGAN/README.md
rename to example/gluon/dc_gan/README.md
diff --git a/example/gluon/DCGAN/__init__.py b/example/gluon/dc_gan/__init__.py
similarity index 100%
rename from example/gluon/DCGAN/__init__.py
rename to example/gluon/dc_gan/__init__.py
diff --git a/example/gluon/DCGAN/dcgan.py b/example/gluon/dc_gan/dcgan.py
similarity index 100%
rename from example/gluon/DCGAN/dcgan.py
rename to example/gluon/dc_gan/dcgan.py
diff --git a/example/gluon/DCGAN/inception_score.py
b/example/gluon/dc_gan/inception_score.py
similarity index 100%
rename from example/gluon/DCGAN/inception_score.py
rename to example/gluon/dc_gan/inception_score.py
diff --git a/example/gluon/kaggle_k_fold_cross_validation.py
b/example/gluon/house_prices/kaggle_k_fold_cross_validation.py
similarity index 100%
rename from example/gluon/kaggle_k_fold_cross_validation.py
rename to example/gluon/house_prices/kaggle_k_fold_cross_validation.py
diff --git a/example/gluon/learning_rate_manipulation.py
b/example/gluon/learning_rate_manipulation.py
deleted file mode 100644
index be1ffc2..000
--- a/example/gluon/learning_rate_manipulation.py
+++ /dev/null
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language go
[incubator-mxnet] branch master updated: Updated / Deleted some examples (#12968)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 3e8a976 Updated / Deleted some examples (#12968)
3e8a976 is described below
commit 3e8a976d805dee130831d4f54b7a5dd9f1a7c7bd
Author: Thomas Delteil
AuthorDate: Fri Nov 2 16:03:51 2018 -0700
Updated / Deleted some examples (#12968)
* Updated / Deleted some examples
* remove onnx test
* remove onnx test
---
ci/docker/runtime_functions.sh | 1 -
example/multivariate_time_series/README.md | 4 +-
example/named_entity_recognition/README.md | 1 -
example/named_entity_recognition/src/metrics.py| 2 +-
example/named_entity_recognition/src/ner.py| 2 +-
example/nce-loss/README.md | 2 +-
example/numpy-ops/numpy_softmax.py | 84 -
example/onnx/super_resolution.py | 86 --
example/python-howto/README.md | 37 --
example/python-howto/data_iter.py | 76 ---
example/python-howto/debug_conv.py | 39 --
example/python-howto/monitor_weights.py| 46
example/python-howto/multiple_outputs.py | 38 --
.../{mxnet_adversarial_vae => vae-gan}/README.md | 0
.../convert_data.py| 0
.../vaegan_mxnet.py| 0
.../python-pytest/onnx/import/onnx_import_test.py | 15
17 files changed, 6 insertions(+), 427 deletions(-)
diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
index 0adec07..095eb57 100755
--- a/ci/docker/runtime_functions.sh
+++ b/ci/docker/runtime_functions.sh
@@ -877,7 +877,6 @@ unittest_centos7_gpu() {
integrationtest_ubuntu_cpu_onnx() {
set -ex
export PYTHONPATH=./python/
- python example/onnx/super_resolution.py
pytest tests/python-pytest/onnx/import/mxnet_backend_test.py
pytest tests/python-pytest/onnx/import/onnx_import_test.py
pytest tests/python-pytest/onnx/import/gluon_backend_test.py
diff --git a/example/multivariate_time_series/README.md
b/example/multivariate_time_series/README.md
index 704c86a..87baca3 100644
--- a/example/multivariate_time_series/README.md
+++ b/example/multivariate_time_series/README.md
@@ -3,6 +3,8 @@
- This repo contains an MXNet implementation of
[this](https://arxiv.org/pdf/1703.07015.pdf) state of the art time series
forecasting model.
- You can find my blog post on the model
[here](https://opringle.github.io/2018/01/05/deep_learning_multivariate_ts.html)
+- A Gluon implementation is available
[here](https://github.com/safrooze/LSTNet-Gluon)
+
## Running the code
@@ -22,7 +24,7 @@
## Hyperparameters
-The default arguements in `lstnet.py` achieve equivolent performance to the
published results. For other datasets, the following hyperparameters provide a
good starting point:
+The default arguements in `lstnet.py` achieve equivalent performance to the
published results. For other datasets, the following hyperparameters provide a
good starting point:
- q = {2^0, 2^1, ... , 2^9} (1 week is typical value)
- Convolutional num filters = {50, 100, 200}
diff --git a/example/named_entity_recognition/README.md
b/example/named_entity_recognition/README.md
index 260c19d..2b28b3b 100644
--- a/example/named_entity_recognition/README.md
+++ b/example/named_entity_recognition/README.md
@@ -11,7 +11,6 @@ To reproduce the preprocessed training data:
1. Download and unzip the data:
https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus/downloads/ner_dataset.csv
2. Move ner_dataset.csv into `./data`
-3. create `./preprocessed_data` directory
3. `$ cd src && python preprocess.py`
To train the model:
diff --git a/example/named_entity_recognition/src/metrics.py
b/example/named_entity_recognition/src/metrics.py
index 40c5015..d3d7378 100644
--- a/example/named_entity_recognition/src/metrics.py
+++ b/example/named_entity_recognition/src/metrics.py
@@ -27,7 +27,7 @@ def load_obj(name):
with open(name + '.pkl', 'rb') as f:
return pickle.load(f)
-tag_dict = load_obj("../preprocessed_data/tag_to_index")
+tag_dict = load_obj("../data/tag_to_index")
not_entity_index = tag_dict["O"]
def classifer_metrics(label, pred):
diff --git a/example/named_entity_recognition/src/ner.py
b/example/named_entity_recognition/src/ner.py
index 561db4c..7f5dd84 100644
--- a/example/named_entity_recognition/src/ner.py
+++ b/example/named_entity_recognition/src/ner.py
@@ -34,7 +34,7 @@ logging.basicConfig(level=logging.DEBUG)
parser = argparse.ArgumentParser(description="Deep neural net
[incubator-mxnet] branch master updated: fix mac r install and windows python build from source docs (#12919)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new ffeaf31 fix mac r install and windows python build from source docs
(#12919)
ffeaf31 is described below
commit ffeaf315a41b1bda48deeaab5a8c4109ea63d7cc
Author: Aaron Markham
AuthorDate: Fri Oct 26 13:15:56 2018 -0700
fix mac r install and windows python build from source docs (#12919)
* fix mac r install and windows python build from source docs
* reorder macos r install instructions
---
docs/install/index.md | 19 ++-
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/docs/install/index.md b/docs/install/index.md
index fc13697..7ddbaa8 100644
--- a/docs/install/index.md
+++ b/docs/install/index.md
@@ -597,7 +597,8 @@ MXNet developers should refer to the MXNet wiki's https://cwiki.apache.
-Install OpenCV and OpenBLAS.
+
+To run MXNet you also should have OpenCV and OpenBLAS installed. You may
install them with `brew` as follows:
```bash
brew install opencv
@@ -757,7 +758,7 @@ All MKL pip packages are experimental prior to version
1.3.0.
-
+
Docker images with *MXNet* are available at [Docker
Hub](https://hub.docker.com/r/mxnet/).
@@ -800,7 +801,13 @@ mxnet/python1.3.0_cpu_mkl deaf9bf61d29
4 days ago
**Step 4** Validate the installation.
-
+
+
+
+
+Refer to the MXNet Windows installation guide
+
+
@@ -886,7 +893,7 @@ Refer to
[#8671](https://github.com/apache/incubator-mxnet/issues/8671) for stat
You can either upgrade your CUDA install or install the MXNet package that
supports your CUDA version.
-
+
@@ -894,7 +901,7 @@ You can either upgrade your CUDA install or install the
MXNet package that suppo
To build from source, refer to the MXNet Windows
installation guide.
-
+
@@ -915,6 +922,8 @@ options(repos = cran)
install.packages("mxnet")
```
+To run MXNet you also should have OpenCV and OpenBLAS installed.
+
[incubator-mxnet] branch master updated: fix cnn visualization tutorial (#12719)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 865255a fix cnn visualization tutorial (#12719)
865255a is described below
commit 865255a398223510d857820f3061a26065bcdb74
Author: Thomas Delteil
AuthorDate: Wed Oct 10 11:31:14 2018 -0700
fix cnn visualization tutorial (#12719)
---
docs/tutorials/vision/cnn_visualization.md | 16 +++-
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/docs/tutorials/vision/cnn_visualization.md
b/docs/tutorials/vision/cnn_visualization.md
index ea027df..940c261 100644
--- a/docs/tutorials/vision/cnn_visualization.md
+++ b/docs/tutorials/vision/cnn_visualization.md
@@ -99,12 +99,18 @@ def get_vgg(num_layers, ctx=mx.cpu(),
root=os.path.join('~', '.mxnet', 'models')
# Get the number of convolution layers and filters
layers, filters = vgg_spec[num_layers]
-# Build the VGG network
+# Build the modified VGG network
net = VGG(layers, filters, **kwargs)
-
-# Load pretrained weights from model zoo
-from mxnet.gluon.model_zoo.model_store import get_model_file
-net.load_params(get_model_file('vgg%d' % num_layers, root=root), ctx=ctx)
+net.initialize(ctx=ctx)
+
+# Get the pretrained model
+vgg = mx.gluon.model_zoo.vision.get_vgg(num_layers, pretrained=True,
ctx=ctx)
+
+# Set the parameters in the new network
+params = vgg.collect_params()
+for key in params:
+param = params[key]
+net.collect_params()[net.prefix+key.replace(vgg.prefix,
'')].set_data(param.data())
return net
[incubator-mxnet] branch master updated: Updated download links and verification instructions (#12651)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 76ae725 Updated download links and verification instructions (#12651) 76ae725 is described below commit 76ae7252a6751275565600d287b7d432b685fd7d Author: Aaron Markham AuthorDate: Mon Sep 24 14:28:11 2018 -0700 Updated download links and verification instructions (#12651) * use mirror link; add verification instructions * use https links --- docs/install/download.md | 56 +--- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/docs/install/download.md b/docs/install/download.md index 3660204..b61b815 100644 --- a/docs/install/download.md +++ b/docs/install/download.md @@ -4,11 +4,51 @@ These source archives are generated from tagged releases. Updates and patches wi | Version | Source | PGP | SHA | |-|-|-|-| -| 1.3.0 | [Download](http://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz) | [Download](http://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.asc) | [Download](http://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.sha512) | -| 1.2.1 | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz.sha512) | -| 1.2.0 | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz.sha512) | -| 1.1.0 | [Download](http://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz.sha512) | -| 1.0.0 | [Download](http://archive.apache.org/dist/incubator/mxnet/1.0.0/apache-mxnet-src-1.0.0-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.0.0/apache-mxnet-src-1.0.0-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/1.0.0/apache-mxnet-src-1.0.0-incubating.tar.gz.sha512) | -| 0.12.1 | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.1/apache-mxnet-src-0.12.1-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.1/apache-mxnet-src-0.12.1-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.1/apache-mxnet-src-0.12.1-incubating.tar.gz.sha512) | -| 0.12.0 | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.0/apache-mxnet-src-0.12.0-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.0/apache-mxnet-src-0.12.0-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.12.0/apache-mxnet-src-0.12.0-incubating.tar.gz.sha512) | -| 0.11.0 | [Download](http://archive.apache.org/dist/incubator/mxnet/0.11.0/apache-mxnet-src-0.11.0-incubating.tar.gz) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.11.0/apache-mxnet-src-0.11.0-incubating.tar.gz.asc) | [Download](http://archive.apache.org/dist/incubator/mxnet/0.11.0/apache-mxnet-src-0.11.0-incubating.tar.gz.sha512) | +| 1.3.0 | [Download](https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz) | [Download](https://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.asc) | [Download](https://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.sha512) | +| 1.2.1 | [Download](https
[incubator-mxnet] branch master updated (d4991a0 -> dba9487)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from d4991a0 [MXNET-360]auto convert str to bytes in img.imdecode when py3 (#10697) add dba9487 Update and modify Windows docs (#12620) No new revisions were added by this update. Summary of changes: docs/install/windows_setup.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[incubator-mxnet] branch master updated: Remove pip overwrites (#12604)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 0f8d2d2 Remove pip overwrites (#12604)
0f8d2d2 is described below
commit 0f8d2d2267880ec575e8d41255f9b02c5d6ad552
Author: Aaron Markham
AuthorDate: Thu Sep 20 10:17:34 2018 -0700
Remove pip overwrites (#12604)
* remove overwrites on pip instructions coming from source
* removing examples version overwrite
---
docs/build_version_doc/AddVersion.py | 25 -
1 file changed, 25 deletions(-)
diff --git a/docs/build_version_doc/AddVersion.py
b/docs/build_version_doc/AddVersion.py
index f5e28df..f429d34 100755
--- a/docs/build_version_doc/AddVersion.py
+++ b/docs/build_version_doc/AddVersion.py
@@ -87,30 +87,5 @@ if __name__ == '__main__':
outstr = outstr.replace('http://mxnet.io',
'https://mxnet.incubator.apache.org/'
'versions/%s' %
(args.current_version))
-# Fix git clone and pip installation to specific tag
-pip_pattern = ['', '-cu80', '-cu75', '-cu80mkl', '-cu75mkl',
'-mkl']
-if args.current_version == 'master':
-outstr = outstr.replace('git clone --recursive
https://github.com/dmlc/mxnet',
-'git clone --recursive
https://github.com/apache/incubator-mxnet.git mxnet')
-for trail in pip_pattern:
-outstr = outstr.replace('pip install mxnet%s<' % (trail),
-'pip install mxnet%s --pre<' %
(trail))
-outstr = outstr.replace('pip install mxnet%s\n<' % (trail),
-'pip install mxnet%s --pre\n<' %
(trail))
-else:
-outstr = outstr.replace('git clone --recursive
https://github.com/dmlc/mxnet',
-'git clone --recursive
https://github.com/apache/incubator-mxnet.git mxnet '
-'--branch %s' % (args.current_version))
-for trail in pip_pattern:
-outstr = outstr.replace('pip install mxnet%s<' % (trail),
-'pip install mxnet%s==%s<' %
(trail, args.current_version))
-outstr = outstr.replace('pip install mxnet%s\n<' % (trail),
-'pip install mxnet%s==%s\n<' %
(trail, args.current_version))
-
-# Add tag for example link
-outstr =
outstr.replace('https://github.com/apache/incubator-mxnet/tree/master/example',
-
'https://github.com/apache/incubator-mxnet/tree/%s/example' %
-(args.current_version))
-
with open(os.path.join(path, name), "w") as outf:
outf.write(outstr)
[incubator-mxnet] branch master updated: improve tutorial redirection (#12607)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new ba993d1 improve tutorial redirection (#12607) ba993d1 is described below commit ba993d1091044ab9e74e84dae21cef340f263bf5 Author: Aaron Markham AuthorDate: Thu Sep 20 09:53:09 2018 -0700 improve tutorial redirection (#12607) --- docs/build_version_doc/artifacts/.htaccess | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/build_version_doc/artifacts/.htaccess b/docs/build_version_doc/artifacts/.htaccess index 95b311a..d553ce5 100644 --- a/docs/build_version_doc/artifacts/.htaccess +++ b/docs/build_version_doc/artifacts/.htaccess @@ -11,7 +11,7 @@ RewriteRule ^versions/[^\/]+/community/.*$ /community/ [R=301,L] RewriteRule ^versions/[^\/]+/faq/.*$ /faq/ [R=301,L] RewriteRule ^versions/[^\/]+/gluon/.*$ /gluon/ [R=301,L] RewriteRule ^versions/[^\/]+/install/.*$ /install/ [R=301,L] -RewriteRule ^versions/[^\/]+/tutorials/.*$ /tutorials/ [R=301,L] +RewriteRule ^versions/[^\/]+/tutorials/(.*)$ /tutorials/$1 [R=301,L] # Redirect navbar APIs that did not exist RewriteRule ^versions/0.11.0/api/python/contrib/onnx.html /error/api.html [R=301,L]
[incubator-mxnet] branch master updated: add TensorRT tutorial to index and fix ToC (#12587)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 73d8897 add TensorRT tutorial to index and fix ToC (#12587) 73d8897 is described below commit 73d88974f8bca1e68441606fb0787a2cd17eb364 Author: Aaron Markham AuthorDate: Wed Sep 19 10:32:22 2018 -0700 add TensorRT tutorial to index and fix ToC (#12587) * adding tensorrt tutorial to index * fix toc for tutorials * adding back new tutorials * add control_flow index to whitelist --- docs/tutorials/control_flow/index.md | 8 docs/tutorials/index.md | 4 +++- tests/tutorials/test_sanity_tutorials.py | 1 + 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/control_flow/index.md b/docs/tutorials/control_flow/index.md new file mode 100644 index 000..87d7289 --- /dev/null +++ b/docs/tutorials/control_flow/index.md @@ -0,0 +1,8 @@ +# Tutorials + +```eval_rst +.. toctree:: + :glob: + + * +``` diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 8a6ac40..df1d892 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -3,9 +3,9 @@ ```eval_rst .. toctree:: :hidden: - basic/index.md c++/index.md + control_flow/index.md embedded/index.md gluon/index.md nlp/index.md @@ -15,6 +15,7 @@ scala/index.md sparse/index.md speech_recognition/index.md + tensorrt/index.md unsupervised_learning/index.md vision/index.md ``` @@ -118,6 +119,7 @@ Select API: * [Large-Scale Multi-Host Multi-GPU Image Classification](/tutorials/vision/large_scale_classification.html) * [Importing an ONNX model into MXNet](/tutorials/onnx/super_resolution.html) * [Hybridize Gluon models with control flows](/tutorials/control_flow/ControlFlowTutorial.html) +* [Optimizing Deep Learning Computation Graphs with TensorRT](/tutorials/tensorrt/inference_with_trt.html) * API Guides * Core APIs * NDArray diff --git a/tests/tutorials/test_sanity_tutorials.py b/tests/tutorials/test_sanity_tutorials.py index 0fa3ccb..f9fb0ac 100644 --- a/tests/tutorials/test_sanity_tutorials.py +++ b/tests/tutorials/test_sanity_tutorials.py @@ -27,6 +27,7 @@ import re whitelist = ['basic/index.md', 'c++/basics.md', 'c++/index.md', + 'control_flow/index.md', 'embedded/index.md', 'embedded/wine_detector.md', 'gluon/index.md',
[incubator-mxnet] branch master updated: Disable installation nightly test (#12571)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new f6d2bef Disable installation nightly test (#12571)
f6d2bef is described below
commit f6d2bef6562116c479b4e615bc9294a8bc404af3
Author: Aaron Markham
AuthorDate: Tue Sep 18 10:07:16 2018 -0700
Disable installation nightly test (#12571)
* force versions to go to API version
* fully disable the nightly installation tests
---
tests/nightly/Jenkinsfile | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/tests/nightly/Jenkinsfile b/tests/nightly/Jenkinsfile
index 35996fb..de550a0 100755
--- a/tests/nightly/Jenkinsfile
+++ b/tests/nightly/Jenkinsfile
@@ -45,7 +45,7 @@ core_logic: {
//Some install guide tests are currently diabled and tracked here:
//1. https://github.com/apache/incubator-mxnet/issues/11369
//2. https://github.com/apache/incubator-mxnet/issues/11288
- utils.docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_virtualenv', false)
+ //utils.docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_virtualenv', false)
//docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_pip', false)
//docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_docker', false)
//docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_source', false)
@@ -59,7 +59,7 @@ core_logic: {
//Some install guide tests are currently diabled and tracked here:
//1. https://github.com/apache/incubator-mxnet/issues/11369
//2. https://github.com/apache/incubator-mxnet/issues/11288
- utils.docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_virtualenv', true)
+ //utils.docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_virtualenv', true)
//docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_pip', true)
//docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_docker', true)
utils.docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_source', true)
@@ -131,4 +131,3 @@ failure_handler: {
}
}
)
-
[incubator-mxnet] branch master updated (9032e93 -> d8c51e5)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 9032e93 Updating news, readme files and bumping master version to 1.3.1 (#12525) add d8c51e5 replacing windows setup with newer instructions (#12504) No new revisions were added by this update. Summary of changes: docs/install/windows_setup.md | 246 -- 1 file changed, 190 insertions(+), 56 deletions(-)
[incubator-mxnet] branch master updated: update apachecon links to https (#12521)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new e81c0e2 update apachecon links to https (#12521) e81c0e2 is described below commit e81c0e246f0bb00a5e0d0b134e93a1af7f0c82bb Author: Aaron Markham AuthorDate: Wed Sep 12 15:53:44 2018 -0700 update apachecon links to https (#12521) * update apachecon links to https * nudging file to get past flakey test --- docs/_static/mxnet-theme/index.html | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/_static/mxnet-theme/index.html b/docs/_static/mxnet-theme/index.html index c8417ef..d23d45d 100644 --- a/docs/_static/mxnet-theme/index.html +++ b/docs/_static/mxnet-theme/index.html @@ -15,7 +15,7 @@ -http://www.apachecon.com/acna18/; class="section-tout-promo">http://www.apachecon.com/acna18/banners/acna-sleek-highres.png; width="65%" alt="apachecon"/> +https://www.apachecon.com/acna18/; class="section-tout-promo">https://www.apachecon.com/acna18/banners/acna-sleek-highres.png; width="65%" alt="apachecon"/> @@ -42,7 +42,6 @@ -
[incubator-mxnet] branch master updated: fix subscribe links, remove disabled icons (#12474)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new acede67 fix subscribe links, remove disabled icons (#12474) acede67 is described below commit acede67649bf40bdbcf0081d4aea0a13aba8f500 Author: Aaron Markham AuthorDate: Mon Sep 10 16:54:17 2018 -0700 fix subscribe links, remove disabled icons (#12474) * fix subscribe links, remove disabled icons * update slack channel --- docs/community/mxnet_channels.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/community/mxnet_channels.md b/docs/community/mxnet_channels.md index 18dc1bc..98cce94 100644 --- a/docs/community/mxnet_channels.md +++ b/docs/community/mxnet_channels.md @@ -2,9 +2,9 @@ Converse with the MXNet community via the following channels: -- [Forum](https://discuss.mxnet.io/): [discuss.mxnet.io](https://discuss.mxnet.io/) -- [MXNet Apache developer mailing list](https://lists.apache.org/list.html?d...@mxnet.apache.org) (d...@mxnet.apache.org): To subscribe, send an email to mailto:user-subscr...@mxnet.apache.org;>dev-subscr...@mxnet.apache.org -- [MXNet Apache user mailing list](https://lists.apache.org/list.html?u...@mxnet.apache.org) (u...@mxnet.apache.org): To subscribe, send an email to mailto:dev-subscr...@mxnet.apache.org;>user-subscr...@mxnet.apache.org -- [MXNet Slack channel](https://apache-mxnet.slack.com): To request an invitation to the channel please subscribe to the mailing list above and then email: mailto:d...@mxnet.apache.org;>d...@mxnet.apache.org +- [Forum](https://discuss.mxnet.io/): [discuss.mxnet.io](https://discuss.mxnet.io/) +- [MXNet Apache developer mailing list](https://lists.apache.org/list.html?d...@mxnet.apache.org) (d...@mxnet.apache.org): To subscribe, send an email to mailto:user-subscr...@mxnet.apache.org;>user-subscr...@mxnet.apache.org +- [MXNet Apache user mailing list](https://lists.apache.org/list.html?u...@mxnet.apache.org) (u...@mxnet.apache.org): To subscribe, send an email to mailto:dev-subscr...@mxnet.apache.org;>dev-subscr...@mxnet.apache.org +- [MXNet Slack channel](https://the-asf.slack.com/) (Channel: #mxnet): To request an invitation to the channel please subscribe to the mailing list above and then email: mailto:d...@mxnet.apache.org;>d...@mxnet.apache.org Note: if you have an email address with apache.org, you do not need an approval to join the MXNet Slack channel.
[incubator-mxnet] branch master updated: Installation instructions consolidation (#12388)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new d7111d3 Installation instructions consolidation (#12388)
d7111d3 is described below
commit d7111d357dd682e9e75c6adc5c73e6b74c5541dd
Author: Aaron Markham
AuthorDate: Wed Sep 5 14:31:42 2018 -0700
Installation instructions consolidation (#12388)
* initial edits for c++ instructions
* consolidation and reorg of install docs
* simplify python section
* fix url rewrite to allow for testing
* further simplify install
* fix render issues
* adjust formatting
* adjust formatting
* fix python/gpu/master
* install footers
* update footers
* add mac dev setup link for osx
* adjust formatting for footer
* separate validation page
* add toc
* adjust formatting
* break out julia and perl
* fix pypi link
* fix formatting
* fix formatting
* fix formatting
* add build from source link
* updating from PR feedback
* add c++ and clojure to ubuntu guide; clarify c++ setup
* added pip chart to install; updates from PR feedback
* correct reference to c infer api
---
docs/_static/js/options.js| 20 +-
docs/install/build_from_source.md | 493 ++--
docs/install/c_plus_plus.md | 29 +
docs/install/centos_setup.md | 98 +-
docs/install/index.md | 2523 -
docs/install/osx_setup.md | 11 +
docs/install/ubuntu_setup.md | 181 +--
docs/install/validate_mxnet.md| 185 +++
docs/install/windows_setup.md | 14 +-
9 files changed, 1093 insertions(+), 2461 deletions(-)
diff --git a/docs/_static/js/options.js b/docs/_static/js/options.js
index 87e50b8..b43f391 100644
--- a/docs/_static/js/options.js
+++ b/docs/_static/js/options.js
@@ -8,7 +8,7 @@ $(document).ready(function () {
function label(lbl) {
return lbl.replace(/[ .]/g, '-').toLowerCase();
}
-
+
function urlSearchParams(searchString) {
let urlDict = new Map();
let searchParams = searchString.substring(1).split("&");
@@ -45,11 +45,11 @@ $(document).ready(function () {
showContent();
if (window.location.href.indexOf("/install/index.html") >= 0) {
if (versionSelect.indexOf(defaultVersion) >= 0) {
-history.pushState(null, null, '/install/index.html?platform='
+ platformSelect + '=' + languageSelect + '=' +
processorSelect);
+history.pushState(null, null, 'index.html?platform=' +
platformSelect + '=' + languageSelect + '=' +
processorSelect);
} else {
-history.pushState(null, null, '/install/index.html?version=' +
versionSelect + '=' + platformSelect + '=' + languageSelect +
'=' + processorSelect);
+history.pushState(null, null, 'index.html?version=' +
versionSelect + '=' + platformSelect + '=' + languageSelect +
'=' + processorSelect);
}
-}
+}
}
function showContent() {
@@ -73,22 +73,22 @@ $(document).ready(function () {
$('.current-version').html( $(this).text() + ' ' );
if ($(this).text().indexOf(defaultVersion) < 0) {
if (window.location.search.indexOf("version") < 0) {
-history.pushState(null, null, '/install/index.html' +
window.location.search.concat( '=' + $(this).text() ));
+history.pushState(null, null, 'index.html' +
window.location.search.concat( '=' + $(this).text() ));
} else {
-history.pushState(null, null, '/install/index.html' +
window.location.search.replace( urlParams.get('version'), $(this).text() ));
+history.pushState(null, null, 'index.html' +
window.location.search.replace( urlParams.get('version'), $(this).text() ));
}
} else if (window.location.search.indexOf("version") >= 0) {
- history.pushState(null, null, '/install/index.html' +
window.location.search.replace( 'version', 'prev' ));
+ history.pushState(null, null, 'index.html' +
window.location.search.replace( 'version', 'prev' ));
}
}
else if ($(this).hasClass("platforms")) {
-history.pushState(null, null, '/install/index.html' +
window.location.search.replace( urlParams.get('platform'), $(this).text() ));
+history.pushState(null, null, 'index.html' +
window.location.search.replace( urlParams.get('platform'), $(this).text() ));
}
else if ($(this).hasClass("lang
[incubator-mxnet] branch master updated: [MXAPPS-581] Straight Dope nightly fixes. (#11934)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c000930 [MXAPPS-581] Straight Dope nightly fixes. (#11934)
c000930 is described below
commit c00093053e1d18f82c6b5109bf56471891986e19
Author: vishaalkapoor <40836875+vishaalkap...@users.noreply.github.com>
AuthorDate: Wed Aug 8 13:31:48 2018 -0700
[MXAPPS-581] Straight Dope nightly fixes. (#11934)
Enable 3 notebooks that were failing tests after making updates to the
Straight Dope book. We also add pandas required by one of these
notebooks.
---
ci/docker/install/ubuntu_nightly_tests.sh | 5 +
.../nightly/straight_dope/test_notebooks_single_gpu.py | 17 +++--
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git a/ci/docker/install/ubuntu_nightly_tests.sh
b/ci/docker/install/ubuntu_nightly_tests.sh
index df56cf5..0e6b437 100755
--- a/ci/docker/install/ubuntu_nightly_tests.sh
+++ b/ci/docker/install/ubuntu_nightly_tests.sh
@@ -30,3 +30,8 @@ apt-get -y install time
# Install for RAT License Check Nightly Test
apt-get install -y subversion maven -y #>/dev/null
+
+# Packages needed for the Straight Dope Nightly tests.
+pip2 install pandas
+pip3 install pandas
+
diff --git a/tests/nightly/straight_dope/test_notebooks_single_gpu.py
b/tests/nightly/straight_dope/test_notebooks_single_gpu.py
index ee7c94c..06ced96 100644
--- a/tests/nightly/straight_dope/test_notebooks_single_gpu.py
+++ b/tests/nightly/straight_dope/test_notebooks_single_gpu.py
@@ -89,9 +89,8 @@ class StraightDopeSingleGpuTests(unittest.TestCase):
def test_probability(self):
assert _test_notebook('chapter01_crashcourse/probability')
-# TODO(vishaalk): Notebook contains the word 'Warning'. Needs to be
updated to a synonym.
-#def test_autograd(self):
-#assert _test_notebook('chapter01_crashcourse/autograd')
+def test_autograd(self):
+assert _test_notebook('chapter01_crashcourse/autograd')
# Chapter 2
@@ -113,9 +112,8 @@ class StraightDopeSingleGpuTests(unittest.TestCase):
def test_regularization_scratch(self):
assert
_test_notebook('chapter02_supervised-learning/regularization-scratch')
-# TODO(vishaalk): Notebook does not appear to be JSON: '{\n "cells": [\n
{\n "cell_type": "m
-#def test_regularization_gluon(self):
-#assert
_test_notebook('chapter02_supervised-learning/regularization-gluon')
+def test_regularization_gluon(self):
+assert
_test_notebook('chapter02_supervised-learning/regularization-gluon')
def test_perceptron(self):
assert _test_notebook('chapter02_supervised-learning/perceptron')
@@ -258,9 +256,8 @@ class StraightDopeSingleGpuTests(unittest.TestCase):
def test_lds_scratch(self):
assert _test_notebook('chapter12_time-series/lds-scratch')
-# TODO(vishaalk): File doesn't appear to be valid JSON.
-#def test_issm_scratch(self):
-#assert _test_notebook('chapter12_time-series/issm-scratch')
+def test_issm_scratch(self):
+assert _test_notebook('chapter12_time-series/issm-scratch')
# Chapter 14
@@ -273,7 +270,7 @@ class StraightDopeSingleGpuTests(unittest.TestCase):
def test_generative_adversarial_networks(self):
assert
_test_notebook('chapter14_generative-adversarial-networks/conditional')
-# TODO(vishaalk): Notebook does not appear to be valid JSON.
+# TODO(vishaalk): Investigate.
#def test_pixel2pixel(self):
#assert
_test_notebook('chapter14_generative-adversarial-networks/pixel2pixel')
[incubator-mxnet] branch master updated: Fix image classification scripts and Improve Fp16 tutorial (#11533)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 54ebc5d Fix image classification scripts and Improve Fp16 tutorial
(#11533)
54ebc5d is described below
commit 54ebc5db7967fe023d987796dd098ecf8a65e507
Author: Rahul Huilgol
AuthorDate: Sat Jul 28 17:26:10 2018 -0700
Fix image classification scripts and Improve Fp16 tutorial (#11533)
* fix bugs and improve tutorial
* improve logging
* update benchmark_score
* Update float16.md
* update link to dmlc web data
* fix train cifar and add random mirroring
* set aug defaults
* fix whitespace
* fix typo
---
docs/faq/float16.md | 12 --
example/gluon/data.py | 15 ++--
example/image-classification/benchmark_score.py | 9 ---
example/image-classification/common/data.py | 31 ++---
example/image-classification/fine-tune.py | 4 ++--
example/image-classification/train_cifar10.py | 8 ++-
example/image-classification/train_imagenet.py | 12 --
src/io/image_aug_default.cc | 2 +-
8 files changed, 66 insertions(+), 27 deletions(-)
diff --git a/docs/faq/float16.md b/docs/faq/float16.md
index cbb308f..b4cd97b 100644
--- a/docs/faq/float16.md
+++ b/docs/faq/float16.md
@@ -102,9 +102,17 @@ python fine-tune.py --network resnet --num-layers 50
--pretrained-model imagenet
```
## Example training results
-Here is a plot to compare the training curves of a Resnet50 v1 network on the
Imagenet 2012 dataset. These training jobs ran for 95 epochs with a batch size
of 1024 using a learning rate of 0.4 decayed by a factor of 1 at epochs
30,60,90 and used Gluon. The only changes made for the float16 job when
compared to the float32 job were that the network and data were cast to
float16, and the multi-precision mode was used for optimizer. The final
accuracies at 95th epoch were **76.598% for flo [...]
+Let us consider training a Resnet50 v1 model on the Imagenet 2012 dataset. For
this model, the GPU memory usage is close to the capacity of V100 GPU with a
batch size of 128 when using float32. Using float16 allows the use of 256 batch
size. Shared below are results using 8 V100 GPUs on a AWS p3.16x large
instance. Let us compare the three scenarios that arise here: float32 with 1024
batch size, float16 with 1024 batch size and float16 with 2048 batch size.
These jobs trained for 90 epoc [...]
-
+Batch size | Data type | Top 1 Validation accuracy | Time to train | Speedup |
+--- | --- | --- | --- | --- |
+1024 | float32 | 76.18% | 11.8 hrs | 1 |
+1024 | float16 | 76.34% | 7.3 hrs | 1.62x |
+2048 | float16 | 76.29% | 6.5 hrs | 1.82x |
+
+
+
+The differences in accuracies above are within normal random variation, and
there is no reason to expect float16 to have better accuracy than float32 in
general. As the plot indicates training behaves similarly for these cases, even
though we didn't have to change any other hyperparameters. We can also see from
the table that using float16 helps train faster through faster computation with
float16 as well as allowing the use of larger batch sizes.
## Things to keep in mind
diff --git a/example/gluon/data.py b/example/gluon/data.py
index 6aa5316..f855c90 100644
--- a/example/gluon/data.py
+++ b/example/gluon/data.py
@@ -21,6 +21,7 @@ import os
import random
import tarfile
import logging
+import tarfile
logging.basicConfig(level=logging.INFO)
import mxnet as mx
@@ -92,9 +93,10 @@ def get_imagenet_iterator(root, batch_size, num_workers,
data_shape=224, dtype='
def get_caltech101_data():
url =
"https://s3.us-east-2.amazonaws.com/mxnet-public/101_ObjectCategories.tar.gz;
dataset_name = "101_ObjectCategories"
-if not os.path.isdir("data"):
+data_folder = "data"
+if not os.path.isdir(data_folder):
os.makedirs(data_folder)
-tar_path = mx.gluon.utils.download(url, path='data')
+tar_path = mx.gluon.utils.download(url, path=data_folder)
if (not os.path.isdir(os.path.join(data_folder, "101_ObjectCategories")) or
not os.path.isdir(os.path.join(data_folder,
"101_ObjectCategories_test"))):
tar = tarfile.open(tar_path, "r:gz")
@@ -110,18 +112,17 @@ def get_caltec
[incubator-mxnet] branch master updated: Corrections to profiling tutorial (#11887)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 196468d Corrections to profiling tutorial (#11887)
196468d is described below
commit 196468dedf3135252ed1a98a3db627ffef019473
Author: Thom Lane
AuthorDate: Sat Jul 28 17:24:42 2018 -0700
Corrections to profiling tutorial (#11887)
Corrected a race condition with stopping profiling. Added mx.nd.waitall to
ensure all operations have completed, including GPU operations that might
otherwise be missing.
Also added alternative code for context selection GPU vs CPU, that had
error before on machines with nvidia-smi.
---
docs/tutorials/python/profiler.md | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/docs/tutorials/python/profiler.md
b/docs/tutorials/python/profiler.md
index 81ecc2a..d99bb19 100644
--- a/docs/tutorials/python/profiler.md
+++ b/docs/tutorials/python/profiler.md
@@ -94,7 +94,10 @@ Let's define a method that will run one training iteration
given data and label.
```python
# Use GPU if available
-ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()
+try:
+mx.test_utils.list_gpus(); ctx = mx.gpu()
+except:
+ctx = mx.cpu()
# Initialize the parameters with random weights
net.collect_params().initialize(mx.init.Xavier(), ctx=ctx)
@@ -144,7 +147,8 @@ profiler.set_state('run')
run_training_iteration(*next(itr))
-# Ask the profiler to stop recording
+# Ask the profiler to stop recording after operations have completed
+mx.nd.waitall()
profiler.set_state('stop')
```
[incubator-mxnet] branch master updated: Add logistic regression tutorial (#11651)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 832a5fb Add logistic regression tutorial (#11651) 832a5fb is described below commit 832a5fbab9ca6e2f13a19abb75b20b845fc86f6e Author: Sergey Sokolov AuthorDate: Wed Jul 25 15:52:38 2018 -0700 Add logistic regression tutorial (#11651) * Add logistic regression tutorial * Code review fix * Add F1 metric, fix code review comments * Add Download buttons script --- .../gluon/logistic_regression_explained.md | 238 + docs/tutorials/index.md| 1 + tests/tutorials/test_tutorials.py | 3 + 3 files changed, 242 insertions(+) diff --git a/docs/tutorials/gluon/logistic_regression_explained.md b/docs/tutorials/gluon/logistic_regression_explained.md new file mode 100644 index 000..8e5e4a5 --- /dev/null +++ b/docs/tutorials/gluon/logistic_regression_explained.md @@ -0,0 +1,238 @@ + +# Logistic regression using Gluon API explained + +Logistic Regression is one of the first models newcomers to Deep Learning are implementing. The focus of this tutorial is to show how to do logistic regression using Gluon API. + +Before anything else, let's import required packages for this tutorial. + + +```python +import numpy as np +import mxnet as mx +from mxnet import nd, autograd, gluon +from mxnet.gluon import nn, Trainer +from mxnet.gluon.data import DataLoader, ArrayDataset + +mx.random.seed(12345) # Added for reproducibility +``` + +In this tutorial we will use fake dataset, which contains 10 features drawn from a normal distribution with mean equals to 0 and standard deviation equals to 1, and a class label, which can be either 0 or 1. The size of the dataset is an arbitrary value. The function below helps us to generate a dataset. Class label `y` is generated via a non-random logic, so the network would have a pattern to look for. Boundary of 3 is selected to make sure that number of positive examples smaller than [...] + + +```python +def get_random_data(size, ctx): +x = nd.normal(0, 1, shape=(size, 10), ctx=ctx) +y = x.sum(axis=1) > 3 +return x, y +``` + +Also, let's define a set of hyperparameters, that we are going to use later. Since our model is simple and dataset is small, we are going to use CPU for calculations. Feel free to change it to GPU for a more advanced scenario. + + +```python +ctx = mx.cpu() +train_data_size = 1000 +val_data_size = 100 +batch_size = 10 +``` + +## Working with data + +To work with data, Apache MXNet provides [Dataset](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.Dataset) and [DataLoader](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.DataLoader) classes. The former is used to provide an indexed access to the data, the latter is used to shuffle and batchify the data. To learn more about working with data in Gluon, please refer to [Gluon Datasets and Dataloaders](https://mxnet.incubator.a [...] + +Below we define training and validation datasets, which we are going to use in the tutorial. + + +```python +train_x, train_ground_truth_class = get_random_data(train_data_size, ctx) +train_dataset = ArrayDataset(train_x, train_ground_truth_class) +train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True) + +val_x, val_ground_truth_class = get_random_data(val_data_size, ctx) +val_dataset = ArrayDataset(val_x, val_ground_truth_class) +val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True) +``` + +## Defining and training the model + +The only requirement for the logistic regression is that the last layer of the network must be a single neuron. Apache MXNet allows us to do so by using [Dense](https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Dense) layer and specifying the number of units to 1. The rest of the network can be arbitrarily complex. + +Below, we define a model which has an input layer of 10 neurons, a couple of inner layers of 10 neurons each, and output layer of 1 neuron. We stack the layers using [HybridSequential](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.HybridSequential) block and initialize parameters of the network using [Xavier](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initialization. + + +```python +net = nn.HybridSequential() + +with net.name_scope(): +net.add(nn.Dense(units=10, activation='relu')) # input layer +net.add(nn.Dense(units=10, activation='relu')) # inner layer 1 +net.add(nn.Dense(units=10, activation='relu')) # inner layer 2 +net.add(nn.Dense(units=1)) # output layer: notice, it must have o
[incubator-mxnet] branch master updated: Fix file name creation for Windows (#11765)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new fa935a8 Fix file name creation for Windows (#11765)
fa935a8 is described below
commit fa935a833f6ebc6581cb399e2ea0a73dd6939520
Author: Sergey Sokolov
AuthorDate: Tue Jul 24 14:57:23 2018 -0700
Fix file name creation for Windows (#11765)
* Fix file name creation for Windows
* Forcing build
* Force build again
---
docs/tutorials/python/predict_image.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/tutorials/python/predict_image.md
b/docs/tutorials/python/predict_image.md
index 3e68be0..a9a0d29 100644
--- a/docs/tutorials/python/predict_image.md
+++ b/docs/tutorials/python/predict_image.md
@@ -57,8 +57,8 @@ from collections import namedtuple
Batch = namedtuple('Batch', ['data'])
def get_image(url, show=False):
-# download and show the image
-fname = mx.test_utils.download(url)
+# download and show the image. Remove query string from the file name.
+fname = mx.test_utils.download(url, fname=url.split('/')[-1].split('?')[0])
img = mx.image.imread(fname)
if img is None:
return None
[incubator-mxnet] branch master updated: [MXAPPS-581] Nightly Straight Dope tests. (#11814)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 64d2e8b [MXAPPS-581] Nightly Straight Dope tests. (#11814)
64d2e8b is described below
commit 64d2e8bfe737fc6268bd4bbb65964b33838f4bd2
Author: vishaalkapoor <40836875+vishaalkap...@users.noreply.github.com>
AuthorDate: Tue Jul 24 13:23:01 2018 -0700
[MXAPPS-581] Nightly Straight Dope tests. (#11814)
* [MXAPPS-581] Nightly Straight Dope tests.
The Straight Dope notebooks will retrieved from the Github repo, run and
scanned for warnings and errors. Because we are not checking accuracy of
the training, we set the number of epochs to 1 to reduce the integration
test run time.
* Common functionality for running and testing notebooks has been
factored into a common test util module.
* Support for running UTF-8 notebooks added (Python2 and 3 compatible).
* Notebooks requiring a single GPU and multi GPUs have been split
into two different test suites so that they can be run on different
hardware.
* Add test to make sure that all notebooks are tested.
* Comment out broken notebooks while they are being fixed (I will
uncomment them in a follow up PR).
* [MXAPPS-581] Download notebooks in test setup.
* Moving logic to download the Straight Dope notebooks to the test
harness.
* Remove cache logic as it is unnecessary.
* [MXAPPS-581] Add a timeout for download of notebooks.
* [MXAPPS-581] Move notebooks requiring multi-gpus.
Move two notebooks requiring multi-GPUs out of the single GPU test suite.
---
ci/docker/runtime_functions.sh | 38 +++
tests/nightly/JenkinsfileForBinaries | 36 +++
tests/nightly/straight_dope/README.md | 7 +
.../straight_dope/straight_dope_test_utils.py | 130
.../straight_dope/test_notebooks_multi_gpu.py | 49 +++
.../straight_dope/test_notebooks_single_gpu.py | 332 +
tests/tutorials/test_tutorials.py | 80 +
tests/utils/notebook_test/__init__.py | 97 ++
8 files changed, 705 insertions(+), 64 deletions(-)
diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
index c899fe5..a0795eb 100755
--- a/ci/docker/runtime_functions.sh
+++ b/ci/docker/runtime_functions.sh
@@ -895,6 +895,44 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+# Nightly 'MXNet: The Straight Dope' Single-GPU Tests
+nightly_straight_dope_python2_single_gpu_tests() {
+set -ex
+cd /work/mxnet/tests/nightly/straight_dope
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TEST_KERNEL=python2
+nosetests-2.7 --with-xunit --xunit-file
nosetests_straight_dope_python2_single_gpu.xml \
+ test_notebooks_single_gpu.py --nologcapture
+}
+
+nightly_straight_dope_python3_single_gpu_tests() {
+set -ex
+cd /work/mxnet/tests/nightly/straight_dope
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TEST_KERNEL=python3
+nosetests-3.4 --with-xunit --xunit-file
nosetests_straight_dope_python3_single_gpu.xml \
+ test_notebooks_single_gpu.py --nologcapture
+}
+
+# Nightly 'MXNet: The Straight Dope' Multi-GPU Tests
+nightly_straight_dope_python2_multi_gpu_tests() {
+set -ex
+cd /work/mxnet/tests/nightly/straight_dope
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TEST_KERNEL=python2
+nosetests-2.7 --with-xunit --xunit-file
nosetests_straight_dope_python2_multi_gpu.xml \
+ test_notebooks_multi_gpu.py --nologcapture
+}
+
+nightly_straight_dope_python3_multi_gpu_tests() {
+set -ex
+cd /work/mxnet/tests/nightly/straight_dope
+export PYTHONPATH=/work/mxnet/python/
+export MXNET_TEST_KERNEL=python3
+nosetests-3.4 --with-xunit --xunit-file
nosetests_straight_dope_python3_multi_gpu.xml \
+ test_notebooks_multi_gpu.py --nologcapture
+}
+
# Deploy
deploy_docs() {
diff --git a/tests/nightly/JenkinsfileForBinaries
b/tests/nightly/JenkinsfileForBinaries
index 3d958b1..0b009d2 100755
--- a/tests/nightly/JenkinsfileForBinaries
+++ b/tests/nightly/JenkinsfileForBinaries
@@ -98,6 +98,42 @@ try {
docker_run('ubuntu_nightly_gpu', 'nightly_test_KVStore_singleNode',
true)
}
}
+},
+'StraightDope: Python2 Single-GPU': {
+ node('mxnetlinux-gpu-p3') {
+ws('workspace/straight_dope-single_gpu') {
+ init_git()
+ unpack_lib('gpu', mx_lib)
+ docker_run('ubuntu_nightly_gpu',
'nightly_straight_dope_python2_single_gpu_tests', true)
+}
+ }
+},
+'StraightDope: Python2 Multi-GPU': {
+ node('mxnetlinux-gpu') {
+ws('wor
[incubator-mxnet] branch master updated: [MXNET-594] Added Learning Rate Finder tutorial (#11304)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new e870890 [MXNET-594] Added Learning Rate Finder tutorial (#11304)
e870890 is described below
commit e870890ab93ae25d51da01a445c965e4db71069b
Author: Thom Lane
AuthorDate: Wed Jul 4 14:13:37 2018 -0700
[MXNET-594] Added Learning Rate Finder tutorial (#11304)
* Added Learning Rate Finder tutorial.
* Updated based on feedback.
* Reverting save_parameters changes.
* Adjusted based on feedback.
* Corrected outdated code comment.
---
docs/tutorials/gluon/learning_rate_finder.md | 325 +++
docs/tutorials/index.md | 1 +
tests/tutorials/test_tutorials.py| 5 +-
3 files changed, 330 insertions(+), 1 deletion(-)
diff --git a/docs/tutorials/gluon/learning_rate_finder.md
b/docs/tutorials/gluon/learning_rate_finder.md
new file mode 100644
index 000..661a017
--- /dev/null
+++ b/docs/tutorials/gluon/learning_rate_finder.md
@@ -0,0 +1,325 @@
+
+# Learning Rate Finder
+
+Setting the learning rate for stochastic gradient descent (SGD) is crucially
important when training neural network because it controls both the speed of
convergence and the ultimate performance of the network. Set the learning too
low and you could be twiddling your thumbs for quite some time as the
parameters update very slowly. Set it too high and the updates will skip over
optimal solutions, or worse the optimizer might not converge at all!
+
+Leslie Smith from the U.S. Naval Research Laboratory presented a method for
finding a good learning rate in a paper called ["Cyclical Learning Rates for
Training Neural Networks"](https://arxiv.org/abs/1506.01186). We implement this
method in MXNet (with the Gluon API) and create a 'Learning Rate Finder' which
you can use while training your own networks. We take a look at the central
idea of the paper, cyclical learning rate schedules, in the ['Advanced Learning
Rate Schedules'](https:/ [...]
+
+## Simple Idea
+
+Given an initialized network, a defined loss and a training dataset we take
the following steps:
+
+1. Train one batch at a time (a.k.a. an iteration)
+2. Start with a very small learning rate (e.g. 0.01) and slowly increase
it every iteration
+3. Record the training loss and continue until we see the training loss diverge
+
+We then analyse the results by plotting a graph of the learning rate against
the training loss as seen below (taking note of the log scales).
+
+https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/lr_finder/finder_plot_w_annotations.png;
width="500px"/>
+
+As expected, for very small learning rates we don't see much change in the
loss as the parameter updates are negligible. At a learning rate of 0.001, we
start to see the loss fall. Setting the initial learning rate here is
reasonable, but we still have the potential to learn faster. We observe a drop
in the loss up until 0.1 where the loss appears to diverge. We want to set the
initial learning rate as high as possible before the loss becomes unstable, so
we choose a learning rate of 0.05.
+
+## Epoch to Iteration
+
+Usually, our unit of work is an epoch (a full pass through the dataset) and
the learning rate would typically be held constant throughout the epoch. With
the Learning Rate Finder (and cyclical learning rate schedules) we are required
to vary the learning rate every iteration. As such we structure our training
code so that a single iteration can be run with a given learning rate. You can
implement Learner as you wish. Just initialize the network, define the loss and
trainer in `__init__` [...]
+
+
+```python
+import mxnet as mx
+
+# Set seed for reproducibility
+mx.random.seed(42)
+
+class Learner():
+def __init__(self, net, data_loader, ctx):
+"""
+:param net: network (mx.gluon.Block)
+:param data_loader: training data loader (mx.gluon.data.DataLoader)
+:param ctx: context (mx.gpu or mx.cpu)
+"""
+self.net = net
+self.data_loader = data_loader
+self.ctx = ctx
+# So we don't need to be in `for batch in data_loader` scope
+# and can call for next batch in `iteration`
+self.data_loader_iter = iter(self.data_loader)
+self.net.initialize(mx.init.Xavier(), ctx=self.ctx)
+self.loss_fn = mx.gluon.loss.SoftmaxCrossEntropyLoss()
+self.trainer = mx.gluon.Trainer(net.collect_params(), 'sgd',
{'learning_rate': .001})
+
+def iteration(self, lr=None, take_step=True):
+"""
+:param lr: learning rate to use for iteration (float)
+:param take_step: take trainer step
[incubator-mxnet] branch master updated: Profiler is already enabled by default in 1.2.0 (#11558)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new eb4b66c Profiler is already enabled by default in 1.2.0 (#11558) eb4b66c is described below commit eb4b66cf90cfd2f5abcece9948fcf174c9827eda Author: Sergey Sokolov AuthorDate: Wed Jul 4 13:15:03 2018 -0700 Profiler is already enabled by default in 1.2.0 (#11558) I have checked that one doesn't need to compile mxnet from source to enable profiling. Installing 1.2.0 version of mxnet is enough to run the code listed in that notebook. I checked it for both CPU and GPU contexts. --- docs/tutorials/python/profiler.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/tutorials/python/profiler.md b/docs/tutorials/python/profiler.md index 46fe23b..81ecc2a 100644 --- a/docs/tutorials/python/profiler.md +++ b/docs/tutorials/python/profiler.md @@ -49,9 +49,7 @@ While it is possible to use `NDArray.waitall()` before and after operations to g ## The correct way to profile -The correct way to measure running time of MXNet models is to use MXNet profiler. In the rest of this tutorial, we will learn how to use the MXNet profiler to measure the running time and memory consumption of MXNet models. - -To use the profiler, you need to build MXNet with `USE_PROFILER=1`. Check the [installation](http://mxnet.incubator.apache.org/install/index.html) page for more information on how to install MXNet from source. After building with `USE_PROFILER=1` and installing, you can import the profiler and configure it from Python code. +The correct way to measure running time of MXNet models is to use MXNet profiler. In the rest of this tutorial, we will learn how to use the MXNet profiler to measure the running time and memory consumption of MXNet models. You can import the profiler and configure it from Python code. ```python from mxnet import profiler
[incubator-mxnet] branch master updated: Enable test_autograd_save_memory (#11437)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new dc291d3 Enable test_autograd_save_memory (#11437)
dc291d3 is described below
commit dc291d3a347c6a062c79132ba124cf989d020c06
Author: Kalyanee Chendke
AuthorDate: Thu Jun 28 13:53:50 2018 -0700
Enable test_autograd_save_memory (#11437)
---
tests/python/gpu/test_operator_gpu.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/tests/python/gpu/test_operator_gpu.py
b/tests/python/gpu/test_operator_gpu.py
index e4b1917..fd76990 100644
--- a/tests/python/gpu/test_operator_gpu.py
+++ b/tests/python/gpu/test_operator_gpu.py
@@ -1628,7 +1628,6 @@ def test_sequence_reverse():
check_sequence_reverse(mx.gpu(0))
-@unittest.skip("Test fails intermittently. Temporarily disabled until fixed.
Tracked at https://github.com/apache/incubator-mxnet/issues/8211;)
@with_seed()
def test_autograd_save_memory():
x = mx.nd.zeros((128, 512, 512), ctx=mx.gpu(0))
[incubator-mxnet] branch master updated: [MXNET-593] Adding 2 tutorials on Learning Rate Schedules (#11296)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new e494cee [MXNET-593] Adding 2 tutorials on Learning Rate Schedules
(#11296)
e494cee is described below
commit e494cee9b540eb19cc0220cc3b3dbfde074b55c9
Author: Thom Lane
AuthorDate: Tue Jun 26 09:54:06 2018 -0700
[MXNET-593] Adding 2 tutorials on Learning Rate Schedules (#11296)
* Added two tutorials on learning rate schedules; basic and advanced.
* Correcting notebook skip line.
* Corrected cosine graph
* Changes based on @KellenSunderland feedback.
---
docs/tutorials/gluon/learning_rate_schedules.md| 328 +
.../gluon/learning_rate_schedules_advanced.md | 308 +++
docs/tutorials/index.md| 2 +
tests/tutorials/test_tutorials.py | 14 +-
4 files changed, 648 insertions(+), 4 deletions(-)
diff --git a/docs/tutorials/gluon/learning_rate_schedules.md
b/docs/tutorials/gluon/learning_rate_schedules.md
new file mode 100644
index 000..dc340b7
--- /dev/null
+++ b/docs/tutorials/gluon/learning_rate_schedules.md
@@ -0,0 +1,328 @@
+
+# Learning Rate Schedules
+
+Setting the learning rate for stochastic gradient descent (SGD) is crucially
important when training neural networks because it controls both the speed of
convergence and the ultimate performance of the network. One of the simplest
learning rate strategies is to have a fixed learning rate throughout the
training process. Choosing a small learning rate allows the optimizer find good
solutions, but this comes at the expense of limiting the initial speed of
convergence. Changing the learnin [...]
+
+Schedules define how the learning rate changes over time and are typically
specified for each epoch or iteration (i.e. batch) of training. Schedules
differ from adaptive methods (such as AdaDelta and Adam) because they:
+
+* change the global learning rate for the optimizer, rather than
parameter-wise learning rates
+* don't take feedback from the training process and are specified beforehand
+
+In this tutorial, we visualize the schedules defined in `mx.lr_scheduler`,
show how to implement custom schedules and see an example of using a schedule
while training models. Since schedules are passed to `mx.optimizer.Optimizer`
classes, these methods work with both Module and Gluon APIs.
+
+
+```python
+%matplotlib inline
+from __future__ import print_function
+import math
+import matplotlib.pyplot as plt
+import mxnet as mx
+from mxnet.gluon import nn
+from mxnet.gluon.data.vision import transforms
+import numpy as np
+```
+
+```python
+def plot_schedule(schedule_fn, iterations=1500):
+# Iteration count starting at 1
+iterations = [i+1 for i in range(iterations)]
+lrs = [schedule_fn(i) for i in iterations]
+plt.scatter(iterations, lrs)
+plt.xlabel("Iteration")
+plt.ylabel("Learning Rate")
+plt.show()
+```
+
+## Schedules
+
+In this section, we take a look at the schedules in `mx.lr_scheduler`. All of
these schedules define the learning rate for a given iteration, and it is
expected that iterations start at 1 rather than 0. So to find the learning rate
for the 100th iteration, you can call `schedule(100)`.
+
+### Stepwise Decay Schedule
+
+One of the most commonly used learning rate schedules is called stepwise
decay, where the learning rate is reduced by a factor at certain intervals.
MXNet implements a `FactorScheduler` for equally spaced intervals, and
`MultiFactorScheduler` for greater control. We start with an example of halving
the learning rate every 250 iterations. More precisely, the learning rate will
be multiplied by `factor` _after_ the `step` index and multiples thereafter. So
in the example below the learning [...]
+
+
+```python
+schedule = mx.lr_scheduler.FactorScheduler(step=250, factor=0.5)
+schedule.base_lr = 1
+plot_schedule(schedule)
+```
+
+
+
+
+
+Note: the `base_lr` is used to determine the initial learning rate. It takes a
default value of 0.01 since we inherit from `mx.lr_scheduler.LRScheduler`, but
it can be set as a property of the schedule. We will see later in this tutorial
that `base_lr` is set automatically when providing the `lr_schedule` to
`Optimizer`. Also be aware that the schedules in `mx.lr_scheduler` have state
(i.e. counters, etc) so calling the schedule out of order may give unexpected
results.
+
+We can define non-uniform intervals with `MultiFactorScheduler` and in the
example below we halve the learning rate _after_ the 250th, 750th (i.e. a step
length of 500 iterations) and 900th (a step length of 150 iterations). As
before, the learning rate of
[incubator-mxnet] branch master updated: [MXNET-521] Add Facebook open-graph tag integration (#11155)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 60d7090 [MXNET-521] Add Facebook open-graph tag integration (#11155)
60d7090 is described below
commit 60d7090bd144de705e403c6990f4855a11a73558
Author: kpmurali <37911926+kpmur...@users.noreply.github.com>
AuthorDate: Wed Jun 13 23:29:32 2018 -0700
[MXNET-521] Add Facebook open-graph tag integration (#11155)
* Adding the first set of Facebook OG tags to layout.html
* Adding the direct link to the Facebook OG image
* Re-arranging the og description tag
* Changes to fix merge conflicts
* Adding secure_url to the og:image tag
* Adding the redundant og:image tag
---
docs/_static/mxnet-theme/layout.html | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/docs/_static/mxnet-theme/layout.html
b/docs/_static/mxnet-theme/layout.html
index 3028342..d72582c 100644
--- a/docs/_static/mxnet-theme/layout.html
+++ b/docs/_static/mxnet-theme/layout.html
@@ -102,11 +102,17 @@
must come *after* these tags. #}
{{ metatags }}
{%- block htmltitle %}
-{%- if pagename != 'index' and 'no title' not in title%}
+
+https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png;
/>
+https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png;
/>
+{%- if pagename != 'index' and 'no title' not in title %}
+
{{ title|striptags|e }}{{ titlesuffix }}
{%- elif pagename == 'index' %}
+
MXNet: A Scalable Deep Learning Framework
{%- else %}
+
{{ pagename.split('/')[0]|capitalize }}{{ titlesuffix }}
{%- endif %}
{%- endblock %}
--
To stop receiving notification emails like this one, please contact
ind...@apache.org.
[incubator-mxnet] branch master updated (fc102c3 -> c54f9f7)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from fc102c3 Fix build.py when CCACHE_DIR is set. (#11226) add c54f9f7 [MXNET-534] Add examples to example list (#11225) No new revisions were added by this update. Summary of changes: example/README.md | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-) -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: Added missing links. (#11067)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 0df58ae Added missing links. (#11067) 0df58ae is described below commit 0df58aef13bc7dcf152ddae3d3f4155e17ac6e51 Author: Thom Lane <thom.e.l...@gmail.com> AuthorDate: Mon May 28 09:39:31 2018 -0700 Added missing links. (#11067) --- docs/tutorials/index.md | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index fce3315..5cdd2d7 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -69,6 +69,7 @@ Select API: * [AutoGrad API with Python control flow](http://gluon-crash-course.mxnet.io/autograd.html) https://upload.wikimedia.org/wikipedia/commons/6/6a/External_link_font_awesome.svg; alt="External link" height="15px" style="margin: 0px 0px 3px 3px;"/> * Data * [Datasets and DataLoaders](/tutorials/gluon/datasets.html) +* [Applying Data Augmentation](/tutorials/gluon/data_augmentation.html) * [Data Augmentation with Masks (for Object Segmentation)](https://mxnet.incubator.apache.org/tutorials/python/data_augmentation_with_masks.html) @@ -89,6 +90,7 @@ Select API: * [Predicting on new images using a pre-trained ImageNet model](/tutorials/python/predict_image.html) * [Fine-Tuning a pre-trained ImageNet model with a new dataset](/faq/finetune.html) * [Large-Scale Multi-Host Multi-GPU Image Classification](/tutorials/vision/large_scale_classification.html) +* [Importing an ONNX model into MXNet](/tutorials/onnx/super_resolution.html) * API Guides * Core APIs * NDArray @@ -105,7 +107,10 @@ Select API: * [Key-Value Store API](/tutorials/python/kvstore.html) * Module APIs * [Module API](/tutorials/basic/module.html) -* [Data Iterators](/tutorials/basic/data.html) +* Data +* [Data Iterators](/tutorials/basic/data.html) +* [Applying Data Augmentation](/tutorials/python/data_augmentation.html) +* [Types of Data Augmentation](/tutorials/python/types_of_data_augmentation.html) -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: Added Module vs Gluon API FAQ (#11038)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 16216a8 Added Module vs Gluon API FAQ (#11038) 16216a8 is described below commit 16216a88806d65c8f38d8e55dc7f219dc369f818 Author: Thom Lane <thom.e.l...@gmail.com> AuthorDate: Thu May 24 16:52:10 2018 -0700 Added Module vs Gluon API FAQ (#11038) --- docs/faq/index.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/faq/index.md b/docs/faq/index.md index 3f42023..ebe16e2 100644 --- a/docs/faq/index.md +++ b/docs/faq/index.md @@ -5,6 +5,9 @@ They also include workflow questions, e.g., how to visualize a neural network co These answers are fairly focused. For more didactic, self-contained introductions to neural networks and full working examples, visit the [tutorials section](../tutorials/index.md). +## API + +* [What's the difference between the Module and Gluon APIs for Python?](http://mxnet.io/api/python/index.html) ## Modeling * [How do I fine-tune pre-trained models to a new dataset?](http://mxnet.io/faq/finetune.html) -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: [MXNET-423] Gluon Model Zoo Pre Trained Model tutorial (#10959)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 6abd654 [MXNET-423] Gluon Model Zoo Pre Trained Model tutorial (#10959) 6abd654 is described below commit 6abd6540af2d5b20ddfe84d086e9b5ff02bb12a2 Author: Thomas Delteil <thomas.delte...@gmail.com> AuthorDate: Thu May 17 21:25:56 2018 -0700 [MXNET-423] Gluon Model Zoo Pre Trained Model tutorial (#10959) * adding the pre-trained model tutorial * adding pretrained model tutorial * updating the tutorial * Update pretrained_models.md * Update test_tutorials.py * Update pretrained_models.md * Update pretrained_models.md * updates following indhu's review * Update pretrained_models.md * Trigger build * Trigger build * Trigger build * implement sina feedback * Update pretrained_models.md * Trigger build --- docs/tutorials/gluon/pretrained_models.md | 375 ++ docs/tutorials/index.md | 2 +- tests/tutorials/test_tutorials.py | 3 + 3 files changed, 379 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/gluon/pretrained_models.md b/docs/tutorials/gluon/pretrained_models.md new file mode 100644 index 000..0de5fdd --- /dev/null +++ b/docs/tutorials/gluon/pretrained_models.md @@ -0,0 +1,375 @@ + +# Using pre-trained models in MXNet + + +In this tutorial we will see how to use multiple pre-trained models with Apache MXNet. First, let's download three image classification models from the Apache MXNet [Gluon model zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html). +* **DenseNet-121** ([research paper](https://arxiv.org/abs/1608.06993)), improved state of the art on [ImageNet dataset](http://image-net.org/challenges/LSVRC) in 2016. +* **MobileNet** ([research paper](https://arxiv.org/abs/1704.04861)), MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks, suitable for mobile applications. +* **ResNet-18** ([research paper](https://arxiv.org/abs/1512.03385v1)), the -152 version is the 2015 winner in multiple categories. + +Why would you want to try multiple models? Why not just pick the one with the best accuracy? As we will see later in the tutorial, even though these models have been trained on the same dataset and optimized for maximum accuracy, they do behave slightly differently on specific images. In addition, prediction speed and memory footprints can vary, and that is an important factor for many applications. By trying a few pretrained models, you have an opportunity to find a model that can be a [...] + + +```python +import json + +import matplotlib.pyplot as plt +import mxnet as mx +from mxnet import gluon, nd +from mxnet.gluon.model_zoo import vision +import numpy as np +%matplotlib inline +``` + +## Loading the model + +The [Gluon Model Zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html) provides a collection of off-the-shelf models. You can get the ImageNet pre-trained model by using `pretrained=True`. +If you want to train on your own classification problem from scratch, you can get an untrained network with a specific number of classes using the `classes` parameter: for example `net = vision.resnet18_v1(classes=10)`. However note that you cannot use the `pretrained` and `classes` parameter at the same time. If you want to use pre-trained weights as initialization of your network except for the last layer, have a look at the last section of this tutorial. + +We can specify the *context* where we want to run the model: the default behavior is to use a CPU context. There are two reasons for this: +* First, this will allow you to test the notebook even if your machine is not equipped with a GPU :) +* Second, we're going to predict a single image and we don't have any specific performance requirements. For production applications where you'd want to predict large batches of images with the best possible throughput, a GPU could definitely be the way to go. +* If you want to use a GPU, make sure you have pip installed the right version of mxnet, or you will get an error when using the `mx.gpu()` context. Refer to the [install instructions](http://mxnet.incubator.apache.org/install/index.html) + + +```python +# We set the context to CPU, you can switch to GPU if you have one and installed a compatible version of MXNet +ctx = mx.cpu() +``` + + +```python +# We can load three the three models +densenet121 = vision.densenet121(pretrained=True, ctx=ctx) +mobileNet = vision.mobilenet0_5(pretrained=True, ctx=ctx) +resnet18 = vision.resnet18_v1(pretrained=True, ctx=ctx) +``` + +We ca
[incubator-mxnet] branch master updated: [MXNET-307] Fix flaky tutorial tests from CI (#10956)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c4144f2 [MXNET-307] Fix flaky tutorial tests from CI (#10956)
c4144f2 is described below
commit c4144f2c8e7377aa9cbc38cf9832ff715149a79d
Author: Thomas Delteil <thomas.delte...@gmail.com>
AuthorDate: Thu May 17 07:35:08 2018 -0700
[MXNET-307] Fix flaky tutorial tests from CI (#10956)
* re-add the predict image tutorial
* Update predict_image.md
* switching to GoogleNet to have a smaller model
* Update inference_on_onnx_model.md
* Update inference_on_onnx_model.md
* Update test_tutorials.py
* Update test_tutorials.py
* Update inference_on_onnx_model.md
* trigger build
* Trigger build
* trigger build
* Trigger build
* update according to review
---
docs/tutorials/index.md| 5 +-
docs/tutorials/onnx/fine_tuning_gluon.md | 22 +++
docs/tutorials/onnx/inference_on_onnx_model.md | 62 +++---
docs/tutorials/python/predict_image.md | 87 +++---
tests/tutorials/test_tutorials.py | 4 +-
5 files changed, 75 insertions(+), 105 deletions(-)
diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md
index f69e1b4..b8f841b 100644
--- a/docs/tutorials/index.md
+++ b/docs/tutorials/index.md
@@ -83,8 +83,9 @@ Select API:
* [Movie Review Classification using Convolutional
Networks](/tutorials/nlp/cnn.html)
* [Generative Adversarial Networks
(GANs)](/tutorials/unsupervised_learning/gan.html)
* [Recommender Systems using Matrix
Factorization](/tutorials/python/matrix_factorization.html)
-* [Speech Recognition with Connectionist Temporal Classification
Loss](https://mxnet.incubator.apache.org/tutorials/speech_recognition/ctc.html)
+* [Speech Recognition with Connectionist Temporal Classification
Loss](/tutorials/speech_recognition/ctc.html)
* Practitioner Guides
+* [Predicting on new images using a pre-trained ImageNet
model](/tutorials/python/predict_image.html)
* [Fine-Tuning a pre-trained ImageNet model with a new
dataset](/faq/finetune.html)
* [Large-Scale Multi-Host Multi-GPU Image
Classification](/tutorials/vision/large_scale_classification.html)
* API Guides
@@ -92,7 +93,7 @@ Select API:
* NDArray
* [NDArray API](/tutorials/gluon/ndarray.html)
* [Advanced NDArray API](/tutorials/basic/ndarray.html)
-* [NDArray
Indexing](https://mxnet.incubator.apache.org/tutorials/basic/ndarray_indexing.html)
+* [NDArray Indexing](/tutorials/basic/ndarray_indexing.html)
* Sparse NDArray
* [Sparse Gradient Updates
(RowSparseNDArray)](/tutorials/sparse/row_sparse.html)
* [Compressed Sparse Row Storage Format
(CSRNDArray)](/tutorials/sparse/csr.html)
diff --git a/docs/tutorials/onnx/fine_tuning_gluon.md
b/docs/tutorials/onnx/fine_tuning_gluon.md
index fc940fc..07d8bdf 100644
--- a/docs/tutorials/onnx/fine_tuning_gluon.md
+++ b/docs/tutorials/onnx/fine_tuning_gluon.md
@@ -40,7 +40,7 @@ logging.basicConfig(level=logging.INFO)
### Downloading supporting files
-These are images and a vizualisation script
+These are images and a vizualisation script:
```python
@@ -59,12 +59,12 @@ from utils import *
## Downloading a model from the ONNX model zoo
-We download a pre-trained model, in our case the
[vgg16](https://arxiv.org/abs/1409.1556) model, trained on
[ImageNet](http://www.image-net.org/) from the [ONNX model
zoo](https://github.com/onnx/models). The model comes packaged in an archive
`tar.gz` file containing an `model.onnx` model file and some sample
input/output data.
+We download a pre-trained model, in our case the
[GoogleNet](https://arxiv.org/abs/1409.4842) model, trained on
[ImageNet](http://www.image-net.org/) from the [ONNX model
zoo](https://github.com/onnx/models). The model comes packaged in an archive
`tar.gz` file containing an `model.onnx` model file.
```python
-base_url = "https://s3.amazonaws.com/download.onnx/models/;
-current_model = "vgg16"
+base_url = "https://s3.amazonaws.com/download.onnx/models/opset_3/;
+current_model = "bvlc_googlenet"
model_folder = "model"
archive_file = "{}.tar.gz".format(current_model)
archive_path = os.path.join(model_folder, archive_file)
@@ -230,15 +230,15 @@ sym.get_internals()
-```
\ No newline at end of file
+
diff --git a/docs/tutorials/onnx/inference_on_onnx_model.md
b/docs/tutorials/onnx/inference_on_onnx_model.md
index 3d4072a..b2522ad 100644
--- a/docs/tutorials/onnx/inference_on_onnx_model.md
+++ b/docs/tutorials/onnx/inference_on_onnx_model.md
@@ -51,12 +51,12 @@ fro
[incubator-mxnet] branch master updated: [MXNET-340] Updated tutorials page. (#10621)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 97511ba [MXNET-340] Updated tutorials page. (#10621) 97511ba is described below commit 97511ba943c436492ae044ae0de2046cd89621bf Author: Thom Lane <thom.e.l...@gmail.com> AuthorDate: Fri May 4 18:54:39 2018 -0700 [MXNET-340] Updated tutorials page. (#10621) * Updated tutorials page. * Combined tutorial links Added "alternative" links. * Corrected typo * Force build. * Force build #2 * Force build #3 * Force #4 --- docs/tutorials/index.md | 346 1 file changed, 113 insertions(+), 233 deletions(-) diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 94ea050..f69e1b4 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -1,269 +1,149 @@ # Tutorials -MXNet has two primary high-level interfaces for its deep learning engine: the Gluon API and the Module API. Tutorials for each are provided below. +MXNet tutorials can be found in this section. A variety of language bindings are available for MXNet (including Python, Scala, C++ and R) and we have a different tutorial section for each language. -`TL;DR:` If you are new to deep learning or MXNet, you should start with the Gluon tutorials. +Are you new to MXNet, and don't have a preference on language? We currently recommend starting with Python, and specifically the Gluon APIs (versus Module APIs) as they're more flexible and easier to debug. -The difference between the two is an imperative versus symbolic programming style. Gluon makes it easy to prototype, build, and train deep learning models without sacrificing training speed by enabling both (1) intuitive imperative Python code development and (2) faster execution by automatically generating a symbolic execution graph using the hybridization feature. +Another great resource for learning MXNet is our [examples section](https://github.com/apache/incubator-mxnet/tree/master/example) which includes a wide variety of models (from basic to state-of-the-art) for a wide variety of tasks including: object detection, style transfer, reinforcement learning, and many others. -The Gluon and Module tutorials are in Python, but you can also find a variety of other MXNet tutorials, such as R, Scala, and C++ in the [Other Languages API Tutorials](#other-mxnet-api-tutorials) section below. + -[Example scripts and applications](#example-scripts-and-applications) as well as [contribution](#contributing-tutorials) info is below. +## Python Tutorials - +We have two types of API available for Python: Gluon APIs and Module APIs. [See here](/api/python/gluon/gluon.html) for a comparison. +A comprehensive introduction to Gluon can be found at [The Straight Dope](http://gluon.mxnet.io/). Structured like a book, it build up from first principles of deep learning and take a theoretical walkthrough of progressively more complex models using the Gluon API. Also check out the [60-Minute Gluon Crash Course](http://gluon-crash-course.mxnet.io/) if you're short on time or have used other deep learning frameworks before. -## Python API Tutorials +Use the tutorial selector below to filter to the relevant tutorials. You might see a download link in the top right corner of some tutorials. Use this to download a Jupyter Notebook version of the tutorial, and re-run and adjust the code as you wish. + + +Select API: Gluon Module - - - - - - Introduction - Applications - - - - - - - - Basics - Neural Networks - Advanced - - - - - - - - - + - - - - - -- [Manipulate data the MXNet way with ndarray](http://gluon.mxnet.io/chapter01_crashcourse/ndarray.html) - -- [Automatic differentiation with autograd](http://gluon.mxnet.io/chapter01_crashcourse/autograd.html) - -- [Linear regression with gluon](http://gluon.mxnet.io/chapter02_supervised-learning/linear-regression-gluon.html) - -- [Serialization - saving, loading and checkpointing](http://gluon.mxnet.io/chapter03_deep-neural-networks/serialization.html) - -- [Gluon Datasets and DataLoaders](http://mxnet.incubator.apache.org/tutorials/gluon/datasets.html) - - - - - - -- [Multilayer perceptrons in gluon](http://gluon.mxnet.io/chapter03_deep-neural-networks/mlp-gluon.html) - -- [Multi-class object detection using CNNs in gluon](http://gluon.mxnet.io/chapter04_convolutional-neural-networks/cnn-gluon.html) - -- [Advanced RNNs with gluon](http://gluon.mxnet.io/chapter05_recurrent-neural-networks/rnns-gluon.html) - - - - - - -- [Plumbing: A look under the hood of gluon](http://gluon.mxnet.io/chapter03_deep-neural-networks/plumbing.html) - -- [Designing a custom layer with gluon](/tutorials/gluon/custom_layer.h
[incubator-mxnet] branch master updated: Update CONTRIBUTORS.md (#10720)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 2e40a16 Update CONTRIBUTORS.md (#10720) 2e40a16 is described below commit 2e40a16ec077e62e0f6bcba462d6683a1088b78e Author: ThomasDelteil <thomas.delte...@gmail.com> AuthorDate: Fri Apr 27 22:17:42 2018 -0700 Update CONTRIBUTORS.md (#10720) * Update CONTRIBUTORS.md * trigger build --- CONTRIBUTORS.md | 5 + 1 file changed, 5 insertions(+) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 9baa24d..5f5302a 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -166,3 +166,8 @@ List of Contributors * [Rahul Huilgol](https://github.com/rahul003) * [Anirudh Subramanian](https://github.com/anirudh2290/) * [Zheqin Wang](https://github.com/rasefon) +* [Thom Lane](https://github.com/thomelane) +* [Sina Afrooze](https://github.com/safrooze) +* [Sergey Sokolov](https://github.com/Ishitori) +* [Thomas Delteil](https://github.com/ThomasDelteil) + -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: [MXNET-307] Extend tutorial instructions (#10658)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 4bf3d01 [MXNET-307] Extend tutorial instructions (#10658) 4bf3d01 is described below commit 4bf3d01d7e3868807f9c562084565c9dd7b0 Author: ThomasDelteil <thomas.delte...@gmail.com> AuthorDate: Wed Apr 25 08:37:21 2018 -0700 [MXNET-307] Extend tutorial instructions (#10658) * extend tutorial instructions * trigger build * trigger build --- example/README.md | 14 ++ 1 file changed, 14 insertions(+) diff --git a/example/README.md b/example/README.md index 5e64126..542162c 100644 --- a/example/README.md +++ b/example/README.md @@ -30,8 +30,14 @@ Example applications or scripts should be submitted in this `example` folder. If you have a tutorial idea for the website, download the [ Jupyter notebook tutorial template](https://github.com/dmlc/mxnet/tree/master/example/MXNetTutorialTemplate.ipynb). + Tutorial location + Notebook tutorials should be submitted in the `docs/tutorials` folder, so that they maybe rendered in the [web site's tutorial section](https://mxnet.incubator.apache.org/tutorials/index.html). +Do not forget to udpdate the `docs/tutorials/index.md` for your tutorial to show up on the website. + + Tutorial formatting + The site expects the format to be markdown, so export your notebook as a .md via the Jupyter web interface menu (File > Download As > Markdown). Then, to enable the download notebook button in the web site's UI ([example](https://mxnet.incubator.apache.org/tutorials/python/linear-regression.html)), add the following as the last line of the file ([example](https://github.com/apache/incubator-mxnet/blame/master/docs/tutorials/python/linear-regression.md#L194)): ``` @@ -44,6 +50,14 @@ If you want some lines to show-up in the markdown but not in the generated noteb ``` Typically when you have a `plt.imshow()` you want the image tag `[png](img.png)` in the `.md` but not in the downloaded notebook as the user will re-generate the plot at run-time. + Tutorial tests + +As part of making sure all our tutorials are running correctly with the latest version of MXNet, each tutorial is run automatically through a python2 and python3 jupyter notebook kernel in the CI, in a GPU environment, checking for errors and warnings. + +Add your own test here `tests/tutorials/test_tutorials.py`. (If you forget, don't worry your PR will not pass the sanity check). + +If your tutorial depends on specific packages, simply add them to this provisionning script: `ci/docker/install/ubuntu_tutorials.sh` + ## List of examples ### Languages Binding Examples -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: Update module.md (#10681)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new d2d7d65 Update module.md (#10681)
d2d7d65 is described below
commit d2d7d657001f8bf5595e3dfd6a099e60bfdfb632
Author: ThomasDelteil <thomas.delte...@gmail.com>
AuthorDate: Wed Apr 25 08:36:04 2018 -0700
Update module.md (#10681)
---
docs/tutorials/basic/module.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/tutorials/basic/module.md b/docs/tutorials/basic/module.md
index dbc7b7a..191e3ba 100644
--- a/docs/tutorials/basic/module.md
+++ b/docs/tutorials/basic/module.md
@@ -44,7 +44,7 @@ import mxnet as mx
import numpy as np
mx.random.seed(1234)
-fname =
mx.test_utils.download('http://archive.ics.uci.edu/ml/machine-learning-databases/letter-recognition/letter-recognition.data')
+fname =
mx.test_utils.download('https://s3.us-east-2.amazonaws.com/mxnet-public/letter_recognition/letter-recognition.data')
data = np.genfromtxt(fname, delimiter=',')[:,1:]
label = np.array([ord(l.split(',')[0])-ord('A') for l in open(fname, 'r')])
--
To stop receiving notification emails like this one, please contact
ind...@apache.org.
[incubator-mxnet] branch master updated: [MXNET-307] Add tutorial tests to the CI (#10608)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 6c3e1a4 [MXNET-307] Add tutorial tests to the CI (#10608)
6c3e1a4 is described below
commit 6c3e1a4ba44ccdceabfd28b933303f4f1a1a0337
Author: ThomasDelteil <thomas.delte...@gmail.com>
AuthorDate: Mon Apr 23 11:20:05 2018 -0700
[MXNET-307] Add tutorial tests to the CI (#10608)
* Initial attempt at linking tutorials in CI
* Update CI to link tutorials
* update
* update typos
* Update ubuntu_scala.sh
trying to find out why sometimes it crashes
* trigger build
* trigger build
* link_tutorial_ci
* fix tutorials
* Update test_tutorials.py
* trigger build
* set seed for test
---
Jenkinsfile| 22 +++
ci/build.py| 27 ++-
ci/docker/Dockerfile.build.ubuntu_gpu | 6 +
ci/docker/install/ubuntu_docs.sh | 2 +-
ci/docker/install/ubuntu_python.sh | 4 +-
ci/docker/install/ubuntu_scala.sh | 8 +-
.../{ubuntu_docs.sh => ubuntu_tutorials.sh}| 8 +-
ci/docker/runtime_functions.sh | 23 +++
docs/build_version_doc/README.md | 2 +
docs/mxdoc.py | 6 +-
docs/tutorials/basic/module.md | 2 -
docs/tutorials/gluon/datasets.md | 23 ++-
docs/tutorials/onnx/fine_tuning_gluon.md | 2 +-
docs/tutorials/onnx/inference_on_onnx_model.md | 2 +-
docs/tutorials/sparse/csr.md | 3 -
docs/tutorials/sparse/train.md | 5 +-
docs/tutorials/unsupervised_learning/gan.md| 36 ++--
.../tutorials/vision/large_scale_classification.md | 12 +-
tests/nightly/Jenkinsfile | 28 ---
tests/nightly/test_tutorial.py | 138 --
tests/nightly/test_tutorial_config.txt | 31
tests/tutorials/test_sanity_tutorials.py | 83 +
tests/tutorials/test_tutorials.py | 200 +
23 files changed, 418 insertions(+), 255 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 6c39736..a4bf2c4 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -722,6 +722,28 @@ try {
}
}
}
+},
+'tutorial tests Python 2 GPU': {
+ node('mxnetlinux-gpu') {
+ws('workspace/it-tutorials-py2') {
+ timeout(time: max_time, unit: 'MINUTES') {
+init_git()
+unpack_lib('gpu')
+sh "ci/build.py --shm-size=3g --nvidiadocker --platform ubuntu_gpu
/work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu"
+ }
+}
+ }
+},
+'tutorial tests Python 3 GPU': {
+ node('mxnetlinux-gpu') {
+ws('workspace/it-tutorials-py3') {
+ timeout(time: max_time, unit: 'MINUTES') {
+init_git()
+unpack_lib('gpu')
+sh "ci/build.py --shm-size=3g --nvidiadocker --platform ubuntu_gpu
/work/runtime_functions.sh tutorialtest_ubuntu_python3_gpu"
+ }
+}
+ }
}
}
diff --git a/ci/build.py b/ci/build.py
index ee36876..6d8d014 100755
--- a/ci/build.py
+++ b/ci/build.py
@@ -89,13 +89,19 @@ def buildir() -> str:
return os.path.join(get_mxnet_root(), "build")
-def container_run(platform: str, docker_binary: str, command: List[str],
dry_run: bool = False, into_container: bool = False) -> str:
+def container_run(platform: str,
+ docker_binary: str,
+ shared_memory_size: str,
+ command: List[str],
+ dry_run: bool = False,
+ into_container: bool = False) -> str:
tag = get_docker_tag(platform)
mx_root = get_mxnet_root()
local_build_folder = buildir()
# We need to create it first, otherwise it will be created by the docker
daemon with root only permissions
os.makedirs(local_build_folder, exist_ok=True)
-runlist = [docker_binary, 'run', '--rm',
+runlist = [docker_binary, 'run', '--rm', '-t',
+'--shm-size={}'.format(shared_memory_size),
'-v', "{}:/work/mxnet".format(mx_root), # mount mxnet root
'-v', "{}:/work/build".format(local_build_folder), # mount mxnet/build
for storing build artifacts
'-u', '{}:{}'.format(os.getuid(), os.getgid()),
@@ -157,6 +163,11 @@ def main() -> int:
help="Use nvidia docker",
action='store_true')
+parser.add_argument("--shm-size",
+
[incubator-mxnet] branch master updated: [MXNET-307] Add .md tutorials to .ipynb for CI integration (#10537)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 2c6d631 [MXNET-307] Add .md tutorials to .ipynb for CI integration
(#10537)
2c6d631 is described below
commit 2c6d631b5ada8b03693942309b972fad0c5d39f1
Author: ThomasDelteil <thomas.delte...@gmail.com>
AuthorDate: Mon Apr 16 16:54:20 2018 -0700
[MXNET-307] Add .md tutorials to .ipynb for CI integration (#10537)
* Updated tutorial to add jupyter notebook download
* Adding tutorials to the CI
* fix python2 error
* update following tholane review
* Update cnn.md
update to latest API
---
docs/tutorials/gluon/customop.md | 3 +
docs/tutorials/gluon/data_augmentation.md | 17 ++-
docs/tutorials/gluon/datasets.md | 91 ++---
docs/tutorials/nlp/cnn.md | 47 ---
docs/tutorials/python/data_augmentation.md | 38 +++---
.../python/data_augmentation_with_masks.md | 20 ++-
docs/tutorials/python/kvstore.md | 141 -
.../tutorials/python/types_of_data_augmentation.md | 47 ---
docs/tutorials/speech_recognition/ctc.md | 11 ++
.../unsupervised_learning/auto_encoders.md | 5 -
docs/tutorials/unsupervised_learning/gan.md| 7 +-
.../tutorials/vision/large_scale_classification.md | 10 ++
tests/nightly/test_tutorial_config.txt | 29 +++--
13 files changed, 262 insertions(+), 204 deletions(-)
diff --git a/docs/tutorials/gluon/customop.md b/docs/tutorials/gluon/customop.md
index e10f398..df10788 100644
--- a/docs/tutorials/gluon/customop.md
+++ b/docs/tutorials/gluon/customop.md
@@ -196,3 +196,6 @@ x = mx.nd.uniform(shape=(4, 3))
y = dense(x)
print(y)
```
+
+
+
diff --git a/docs/tutorials/gluon/data_augmentation.md
b/docs/tutorials/gluon/data_augmentation.md
index 97bb46f..df7462c 100644
--- a/docs/tutorials/gluon/data_augmentation.md
+++ b/docs/tutorials/gluon/data_augmentation.md
@@ -10,6 +10,7 @@ import mxnet as mx # used version '1.0.0' at time of writing
import numpy as np
from matplotlib.pyplot import imshow
import multiprocessing
+import os
mx.random.seed(42) # set seed for repeatability
```
@@ -27,17 +28,17 @@ def plot_mx_array(array):
```
```python
-!mkdir -p data/images
-!wget
https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg
-P ./data/images/
+image_folder = os.path.join('data','images')
+mx.test_utils.download('https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg',
dirname=image_folder)
```
```python
-example_image = mx.image.imread("./data/images/0.jpg").astype("float32")
+example_image = mx.image.imread(os.path.join(image_folder,
"0.jpg")).astype("float32")
plot_mx_array(example_image)
```
-
+
## Quick start with
[`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset)
@@ -61,7 +62,7 @@ def aug_transform(data, label):
return data, label
-training_dataset = mx.gluon.data.vision.ImageFolderDataset('./data',
transform=aug_transform)
+training_dataset = mx.gluon.data.vision.ImageFolderDataset('data',
transform=aug_transform)
```
@@ -75,7 +76,7 @@ plot_mx_array(sample_data*255)
```
-
+
In practice you should load images from a dataset with a
[`mxnet.gluon.data.DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader)
to take advantage of automatic batching and shuffling. Under the hood the
`DataLoader` calls `__getitem__`, but you shouldn't need to call directly for
anything other than debugging. Some practitioners pre-augment their datasets by
applying a fixed number of augmentations to each image and [...]
@@ -93,4 +94,6 @@ for data_batch, label_batch in training_data_loader:
```
-
\ No newline at end of file
+
+
+
\ No newline at end of file
diff --git a/doc
[incubator-mxnet] branch master updated: [MXNET-307] Add tutorials to the CI + Fix them (#10495)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new dd0ff7a [MXNET-307] Add tutorials to the CI + Fix them (#10495)
dd0ff7a is described below
commit dd0ff7a9bb2a83a5be5f41575cc8660b2938682c
Author: ThomasDelteil <thomas.delte...@gmail.com>
AuthorDate: Thu Apr 12 10:45:39 2018 -0700
[MXNET-307] Add tutorials to the CI + Fix them (#10495)
* update ci init
* Fix MNIST module tutorial
* fixing predict image tutorial
* fixing basic/data tutorial
* removed old tutorials
* fixed ndarray_indexing
* Fix matrix factorization
* Fix gluon mnist
* fix hybrid
* Final fixes
* Update data.md
* Update test_tutorial.py
* Update matrix_factorization.md
---
.gitignore| 1 +
docs/tutorials/basic/data.md | 36 ---
docs/tutorials/basic/image_io.md | 99 ---
docs/tutorials/basic/ndarray_indexing.md | 4 +-
docs/tutorials/basic/record_io.md | 129
docs/tutorials/gluon/hybrid.md| 6 +-
docs/tutorials/gluon/mnist.md | 33 ---
docs/tutorials/python/linear-regression.md| 8 +-
docs/tutorials/python/matrix_factorization.md | 59 ++-
docs/tutorials/python/mnist.md| 13 ++-
docs/tutorials/python/predict_image.md| 18 ++--
docs/tutorials/sparse/row_sparse.md | 12 +--
docs/tutorials/sparse/train.md| 15 +--
tests/nightly/test_tutorial.py| 137 +++---
tests/nightly/test_tutorial_config.txt| 13 +++
15 files changed, 214 insertions(+), 369 deletions(-)
diff --git a/.gitignore b/.gitignore
index 6be98c5..d585672 100644
--- a/.gitignore
+++ b/.gitignore
@@ -136,6 +136,7 @@ lib
#Notebook Automated Test
!tests/nightly/test_tutorial_config.txt
!tests/nightly/TestNotebook
+tests/nightly/tmp_notebook
# pip building tools
tools/pip_package/build
diff --git a/docs/tutorials/basic/data.md b/docs/tutorials/basic/data.md
index 54ee334..fdd5015 100644
--- a/docs/tutorials/basic/data.md
+++ b/docs/tutorials/basic/data.md
@@ -15,12 +15,6 @@ To complete this tutorial, we need:
```
$ pip install opencv-python requests matplotlib jupyter
```
-- Set the environment variable `MXNET_HOME` to the root of the MXNet source
folder.
-
-```
-$ git clone https://github.com/dmlc/mxnet ~/mxnet
-$ export MXNET_HOME='~/mxnet'
-```
## MXNet Data Iterator
Data Iterators in *MXNet* are similar to Python iterator objects.
@@ -61,6 +55,11 @@ we can use the
[__`NDArrayIter`__](http://mxnet.io/api/python/io/io.html#mxnet.i
```python
import numpy as np
+
+# fix the seed
+np.random.seed(42)
+mx.random.seed(42)
+
data = np.random.rand(100,3)
label = np.random.randint(0, 10, (100,))
data_iter = mx.io.NDArrayIter(data=data, label=label, batch_size=30)
@@ -341,7 +340,7 @@ Let's download sample images that we can work with.
```python
fname =
mx.test_utils.download(url='http://data.mxnet.io/data/test_images.tar.gz',
dirname='data', overwrite=False)
tar = tarfile.open(fname)
-tar.extractall(path='./data')
+tar.extractall(path=os.path.join('.','data'))
tar.close()
```
@@ -351,7 +350,7 @@ tar.close()
**Note:** You will still need ``OpenCV``(not the CV2 Python library) installed
to use `mx.image.imdecode`.
```python
-img = mx.image.imdecode(open('data/test_images/ILSVRC2012_val_0001.JPEG',
'rb').read())
+img =
mx.image.imdecode(open(os.path.join('data','test_images','ILSVRC2012_val_0001.JPEG'),
'rb').read())
plt.imshow(img.asnumpy()); plt.show()
```
@@ -382,7 +381,7 @@ Download and unzip
```python
fname =
mx.test_utils.download(url='http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz',
dirname='data', overwrite=False)
tar = tarfile.open(fname)
-tar.extractall(path='./data')
+tar.extractall(path=os.path.join('.','data'))
tar.close()
```
@@ -392,7 +391,14 @@ Now let's convert them into record io format using the
`im2rec.py` utility scrip
First, we need to make a list that contains all the image files and their
categories:
```python
-os.system('python %s/tools/im2rec.py --list --recursive --test-ratio=0.2
data/caltech data/101_ObjectCategories'%os.environ['MXNET_HOME'])
+mxnet_path = os.path.dirname(mx.__file__)
+im2rec_path = os.path.join(mxnet_path, 'tools','im2rec.py')
+data_path = os.path.join('data','101_ObjectCategories')
+prefix_path = os.path.join('data','caltech')
+
+with open(os.devnull, 'wb') as devnull:
+subprocess.check_call(['python', im2rec_path, '--list', '--recursive',
'--test-ratio=0.2', prefix_path, data_path],
+ stdout=d
[incubator-mxnet] branch master updated: [MXNET-102] Added tutorial on how to use data augmenters. (#10055)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 4c803e9 [MXNET-102] Added tutorial on how to use data augmenters.
(#10055)
4c803e9 is described below
commit 4c803e9cca1dea113f36e1c28f2188d3e0fda34c
Author: Thom Lane <thom.e.l...@gmail.com>
AuthorDate: Thu Apr 5 15:27:27 2018 -0700
[MXNET-102] Added tutorial on how to use data augmenters. (#10055)
* [MXNET-102] Added tutorial on how to use data augmenters.
* Removed Gluon from tutorial.
* Added Gluon version.
* Updated index.md
* Commit to force build.
---
docs/tutorials/gluon/data_augmentation.md | 96 ++
docs/tutorials/index.md| 5 +
docs/tutorials/python/data_augmentation.md | 155 +
3 files changed, 256 insertions(+)
diff --git a/docs/tutorials/gluon/data_augmentation.md
b/docs/tutorials/gluon/data_augmentation.md
new file mode 100644
index 000..97bb46f
--- /dev/null
+++ b/docs/tutorials/gluon/data_augmentation.md
@@ -0,0 +1,96 @@
+# Methods of applying data augmentation (Gluon API)
+
+Data Augmentation is a regularization technique that's used to avoid
overfitting when training Machine Learning models. Although the technique can
be applied in a variety of domains, it's very common in Computer Vision.
Adjustments are made to the original images in the training dataset before
being used in training. Some example adjustments include translating, cropping,
scaling, rotating, changing brightness and contrast. We do this to reduce the
dependence of the model on spurious cha [...]
+
+In this tutorial we demonstrate a method of applying data augmentation with
Gluon
[`mxnet.gluon.data.Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.Dataset)s,
specifically the
[`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset).
+
+```python
+%matplotlib inline
+import mxnet as mx # used version '1.0.0' at time of writing
+import numpy as np
+from matplotlib.pyplot import imshow
+import multiprocessing
+
+mx.random.seed(42) # set seed for repeatability
+```
+
+We define a utility function below, that will be used for visualising the
augmentations in the tutorial.
+
+
+```python
+def plot_mx_array(array):
+"""
+Array expected to be height x width x 3 (channels), and values are floats
between 0 and 255.
+"""
+assert array.shape[2] == 3, "RGB Channel should be last"
+imshow((array.clip(0, 255)/255).asnumpy())
+```
+
+```python
+!mkdir -p data/images
+!wget
https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg
-P ./data/images/
+```
+
+```python
+example_image = mx.image.imread("./data/images/0.jpg").astype("float32")
+plot_mx_array(example_image)
+```
+
+
+
+
+
+## Quick start with
[`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset)
+
+Using Gluon, it's simple to add data augmentation to your training pipeline.
When creating either
[`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset)
or
[`ImageRecordDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageRecordDataset),
you can pass a `transform` function that will be applied to each image in the
dataset, every time it's loaded from [...]
+
+We define `aug_transform` below to perform a selection of augmentation steps
and pass it to our dataset. It's worth noting that augmentations should only be
applied to the training data (and not the test data), so you don't want to pass
this augmentation transform function to the testing dataset.
+
+[`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter)
is a useful function for creating a diverse set of augmentations at once.
Despite the singular `CreateAugmenter`, this function actually returns a list
of Augmenters. We can then loop through this list and apply each type of
augmentation one after another. Although the parameters of `CreateAugmenter`
are fixed, the random augmentations (suc [...]
+
+
+```python
+def aug_transform(data, label):
+data = data.astype('float32')/255
+augs = mx.image.CreateAugmenter(data_shape=(3, 300, 300),
+rand_crop=0.5, rand_mirror=True,
inter_method=10,
+
[incubator-mxnet] branch master updated: Add CNN - character level to the examples list (#10372)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 74cd220 Add CNN - character level to the examples list (#10372) 74cd220 is described below commit 74cd22011f6836f129f6506d50d6e293eaa7 Author: ThomasDelteil <thomas.delte...@gmail.com> AuthorDate: Thu Apr 5 13:04:32 2018 -0700 Add CNN - character level to the examples list (#10372) --- example/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/example/README.md b/example/README.md index 1ad66e9..5e64126 100644 --- a/example/README.md +++ b/example/README.md @@ -150,6 +150,7 @@ Typically when you have a `plt.imshow()` you want the image tag `[png](img.png)` * [class active maps](https://github.com/dmlc/mxnet-notebooks/blob/master/python/moved-from-mxnet/class_active_maps.ipynb) - A demo of how to localize the discriminative regions in an image using global average pooling (GAP) in CNNs. * [DMLC MXNet Notebooks](https://github.com/dmlc/mxnet-notebooks) DMLC's repo for various notebooks ranging from basic usages of MXNet to state-of-the-art deep learning applications. * [AWS Seoul Summit 2017 Demos](https://github.com/sxjscience/aws-summit-2017-seoul) The demo codes and ipython notebooks in AWS Seoul Summit 2017. +* [Character-level CNN for text classification](https://github.com/ThomasDelteil/CNN_NLP_MXNet) Performing category classification on Amazon reviews using Gluon and character-level Convolutional Neural Networks ### Mobile App Examples --- -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: [MXNET-242][Tutorial] Fine-tuning ONNX model in Gluon (#10283)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 224b0c9 [MXNET-242][Tutorial] Fine-tuning ONNX model in Gluon (#10283) 224b0c9 is described below commit 224b0c9fc9a9bc733e70199509dc177d62f71aaa Author: ThomasDelteil <thomas.delte...@gmail.com> AuthorDate: Mon Apr 2 15:24:55 2018 -0700 [MXNET-242][Tutorial] Fine-tuning ONNX model in Gluon (#10283) * ONNX fine-tuning tutorial * Update inference onnx notebook with feedback * Update inference_on_onnx_model.md * Update fine_tuning_gluon.md * adding the onnx tutorial --- docs/api/python/contrib/onnx.md| 1 + docs/tutorials/index.md| 77 ++--- docs/tutorials/onnx/fine_tuning_gluon.md | 433 + docs/tutorials/onnx/inference_on_onnx_model.md | 53 ++- 4 files changed, 492 insertions(+), 72 deletions(-) diff --git a/docs/api/python/contrib/onnx.md b/docs/api/python/contrib/onnx.md index 2a8f1fc..44aabaf 100644 --- a/docs/api/python/contrib/onnx.md +++ b/docs/api/python/contrib/onnx.md @@ -33,6 +33,7 @@ This document describes all the ONNX-MXNet APIs. /tutorials/onnx/super_resolution.md /tutorials/onnx/inference_on_onnx_model.md + /tutorials/onnx/fine_tuning_gluon.md ``` ## API Reference diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 62cb5eb..4aba4a6 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -111,20 +111,21 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of -- [Creating custom operators with numpy](http://mxnet.incubator.apache.org/tutorials/gluon/customop.html) +- [Creating custom operators with numpy](/tutorials/gluon/customop.html) -- [Handwritten digit recognition (MNIST)](http://mxnet.incubator.apache.org/tutorials/gluon/mnist.html) +- [Handwritten digit recognition (MNIST)](/tutorials/gluon/mnist.html) -- [Hybrid network example](http://mxnet.incubator.apache.org/tutorials/gluon/hybrid.html) +- [Hybrid network example](/tutorials/gluon/hybrid.html) -- [Neural network building blocks with gluon](http://mxnet.incubator.apache.org/tutorials/gluon/gluon.html) +- [Neural network building blocks with gluon](/tutorials/gluon/gluon.html) -- [Simple autograd example](http://mxnet.incubator.apache.org/tutorials/gluon/autograd.html) +- [Simple autograd example](/tutorials/gluon/autograd.html) -- [Data Augmentation with Masks (for Object Segmentation)](http://mxnet.incubator.apache.org/tutorials/python/data_augmentation_with_masks.html) +- [Data Augmentation with Masks (for Object Segmentation)](/tutorials/python/data_augmentation_with_masks.html) -- [Inference using an ONNX model](http://mxnet.incubator.apache.org/tutorials/onnx/inference_on_onnx_model.html) +- [Inference using an ONNX model](/tutorials/onnx/inference_on_onnx_model.html) +- [Fine-tuning an ONNX model on Gluon](/tutorials/onnx/fine_tuning_gluon.html) @@ -138,42 +139,42 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of -- [Imperative tensor operations on CPU/GPU](http://mxnet.incubator.apache.org/tutorials/basic/ndarray.html) +- [Imperative tensor operations on CPU/GPU](/tutorials/basic/ndarray.html) -- [NDArray Indexing](../tutorials/basic/ndarray_indexing.html) +- [NDArray Indexing](/tutorials/basic/ndarray_indexing.html) -- [Symbol API](http://mxnet.incubator.apache.org/tutorials/basic/symbol.html) +- [Symbol API](/tutorials/basic/symbol.html) -- [Module API](http://mxnet.incubator.apache.org/tutorials/basic/module.html) +- [Module API](/tutorials/basic/module.html) -- [Iterators - Loading data](http://mxnet.incubator.apache.org/tutorials/basic/data.html) +- [Iterators - Loading data](/tutorials/basic/data.html) -- [Linear regression](http://mxnet.incubator.apache.org/tutorials/python/linear-regression.html) +- [Linear regression](/tutorials/python/linear-regression.html) -- [MNIST - handwriting recognition](http://mxnet.incubator.apache.org/tutorials/python/mnist.html) +- [MNIST - handwriting recognition](/tutorials/python/mnist.html) -- [Large scale image classification](http://mxnet.incubator.apache.org/tutorials/vision/large_scale_classification.html) +- [Large scale image classification](/tutorials/vision/large_scale_classification.html) -- [NDArray in Compressed Sparse Row storage format](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) +- [NDArray in Compressed Sparse Row storage format](/tutorials/sparse/csr.html) -- [Sparse gradient updates](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html) +- [Sparse gradient updates](/tutorials/sparse/row_sparse.html) -- [Train a linear regression model with
[incubator-mxnet] branch master updated: [MXNET-103] Added tutorial on types of data augmentations. (#10039)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new fe6220b [MXNET-103] Added tutorial on types of data augmentations.
(#10039)
fe6220b is described below
commit fe6220b66d2079f27bdc23823595304abd004bfc
Author: Thom Lane <thom.e.l...@gmail.com>
AuthorDate: Mon Apr 2 11:11:39 2018 -0700
[MXNET-103] Added tutorial on types of data augmentations. (#10039)
* Added tutorial on types of data augmentations. With images to demonstrate
effects.
* Removed Gluon references.
Moved custom augmentation to FAQ.
* Added to index.md
---
docs/tutorials/index.md| 2 +
.../tutorials/python/types_of_data_augmentation.md | 379 +
2 files changed, 381 insertions(+)
diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md
index b254cdd..62cb5eb 100644
--- a/docs/tutorials/index.md
+++ b/docs/tutorials/index.md
@@ -173,6 +173,8 @@ The Gluon and Module tutorials are in Python, but you can
also find a variety of
- [Train a linear regression model with sparse
symbols](http://mxnet.incubator.apache.org/tutorials/sparse/train.html)
+- [Types of Data
Augmentation](http://mxnet.incubator.apache.org/tutorials/python/types_of_data_augmentation.html)
+
diff --git a/docs/tutorials/python/types_of_data_augmentation.md
b/docs/tutorials/python/types_of_data_augmentation.md
new file mode 100644
index 000..9ace047
--- /dev/null
+++ b/docs/tutorials/python/types_of_data_augmentation.md
@@ -0,0 +1,379 @@
+
+# Types of Data Augmentation
+
+Data Augmentation is a regularization technique that's used to avoid
overfitting when training Machine Learning models. Although the technique can
be applied in a variety of domains, it's very common in Computer Vision, and
this will be the focus of the tutorial.
+
+Adjustments are made to the original images in the training dataset before
being used in training. Some example adjustments include translating, croping,
scaling, rotating, changing brightness and contrast. We do this to reduce the
dependence of the model on spurious characteristics; e.g. training data may
only contain faces that fill 1/4 of the image, so the model trainied without
data augmentation might unhelpfully learn that faces can only be of this size.
+
+After defining some utility functions to visualise the example images, this
tutorial details each different augmentation that can be used to adjust both
the position and the colors of images. We discuss augmentations that are
combined into single functions, and conclude with a FAQ section.
+
+
+```python
+%matplotlib inline
+from matplotlib.pyplot import imshow
+import mxnet as mx # used version '1.0.0' at time of writing
+import numpy as np
+
+mx.random.seed(42) # set seed for repeatability
+```
+
+We define a utility function below, that will be used for visualising the
augmentations in the tutorial.
+
+
+```python
+def plot_mx_array(array):
+"""
+Array expected to be height x width x 3 (channels), and values are floats
between 0 and 255.
+"""
+assert array.shape[2] == 3, "RGB Channel should be last"
+imshow((array.clip(0, 255)/255).asnumpy())
+```
+
+We load an example image, this will be the target for our augmentations in the
tutorial.
+
+```python
+!wget
https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg
+```
+
+```python
+example_image = mx.image.imread("./0.jpg")
+assert str(example_image.dtype) == ""
+```
+
+
+You'll notice that the image is loaded with with `numpy.int8` datatype. Some
functions such as `swapaxes` don't work on `int` types, so we'll convert to
`float32`, and visualize.
+
+
+```python
+example_image = example_image.astype("float32")
+plot_mx_array(example_image)
+```
+
+
+
+
+
+# Position Augmentation
+
+One form of augmentation affects the position of pixel values. Using
combinations of slicing, scaling, translating, rotating and fliping the values
of the original image can be shifted to create new images. Some operations
(like scaling and rotation) require interpolation as pixels in the new image
are combinations of pixels in the original image.
+
+### Crop
+
+You can use
[`mxnet.image.RandomCropAug`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=randomcropaug#mxnet.image.RandomCropAug)
and
[`mxnet.image.CenterCropAug`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=centercropaug#mxnet.image.CenterCropAug)
to create instances of the Augmenter class, which can be called just like a
function.
[incubator-mxnet] branch master updated: [MXNET-141] Add tutorial Gluon Datasets and DataLoaders (#10251)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 9a0d002 [MXNET-141] Add tutorial Gluon Datasets and DataLoaders (#10251) 9a0d002 is described below commit 9a0d0028695cbc3553ffadf3537b1003bb0006c4 Author: Thom Lane <thom.e.l...@gmail.com> AuthorDate: Mon Apr 2 09:53:43 2018 -0700 [MXNET-141] Add tutorial Gluon Datasets and DataLoaders (#10251) * Added tutorial for Gluon datasets and data loaders. * Changes as per code review. * Added link to tutorial in index.md. * Cut section on RecordIO. Moved num_workers discussion higher up. Removed Gluon DataLoader to Module DataIter wrapper. --- docs/tutorials/gluon/datasets.md | 310 +++ docs/tutorials/index.md | 2 + 2 files changed, 312 insertions(+) diff --git a/docs/tutorials/gluon/datasets.md b/docs/tutorials/gluon/datasets.md new file mode 100644 index 000..248ea02 --- /dev/null +++ b/docs/tutorials/gluon/datasets.md @@ -0,0 +1,310 @@ + +# Gluon `Dataset`s and `DataLoader` + +One of the most critical steps for model training and inference is loading the data: without data you can't do Machine Learning! In this tutorial we use the Gluon API to define a [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) and use a [`DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader) to iterate through the dataset in mini-batches. + +## Introduction to `Dataset`s + +[`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) objects are used to represent collections of data, and include methods to load and parse the data (that is often stored on disk). Gluon has a number of different [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) classes for working with image data straight out-of-the-box, but we'll use the [`ArrayDataset` [...] + +We first start by generating random data `X` (with 3 variables) and corresponding random labels `y` to simulate a typical supervised learning task. We generate 10 samples and we pass them all to the [`ArrayDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=arraydataset#mxnet.gluon.data.ArrayDataset). + + +```python +import mxnet as mx + +X = mx.random.uniform(shape=(10, 3)) +y = mx.random.uniform(shape=(10, 1)) +dataset = mx.gluon.data.dataset.ArrayDataset(X, y) +``` + +A key feature of a [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) is the __*ability to retrieve a single sample given an index*__. Our random data and labels were generated in memory, so this [`ArrayDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=arraydataset#mxnet.gluon.data.ArrayDataset) doesn't have to load anything from disk, but the interface is the same for all [`Dataset`](https [...] + + +```python +sample_idx = 4 +sample = dataset[sample_idx] + +assert len(sample) == 2 +assert sample[0].shape == (3, ) +assert sample[1].shape == (1, ) +print(sample) +``` + +( + [ 0.4375872 0.29753461 0.89177299] + , + [ 0.83261985] + ) + + +We get a tuple of a data sample and its corresponding label, which makes sense because we passed the data `X` and the labels `y` in that order when we instantiated the [`ArrayDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=arraydataset#mxnet.gluon.data.ArrayDataset). We don't usually retrieve individual samples from [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset) objects though (unless [...] + +## Introduction to `DataLoader` + +A [`DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader) is used to create mini-batches of samples from a [`Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataset#mxnet.gluon.data.Dataset), and provides a convenient iterator interface for looping these batches. It's typically much more efficient to pass a mini-batch of data through a neural network than a single sample at a time, be [...] + +Another benefit of using [`DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader) is the ability to easily load data in parallel using [`multiprocessing`](https://docs.python.org/3.6/library/multiprocessing.html). Just set the `num_workers` parameter to the number of CPUs avaliable on your machine for m
[incubator-mxnet] branch master updated: [MXNET-118] Added tutorial for Data Augmentation with Masks (#10178)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new c69a071 [MXNET-118] Added tutorial for Data Augmentation with Masks (#10178) c69a071 is described below commit c69a071e891fe7c5358bcc048bbd5ab86a928df5 Author: Thom Lane <thom.e.l...@gmail.com> AuthorDate: Fri Mar 30 16:54:02 2018 -0700 [MXNET-118] Added tutorial for Data Augmentation with Masks (#10178) * Added tutorial on Data Augmentation with Masks (for Object Segmentation). * Added reference to new tutorial in tutorials index. Added wget for downloading sample images. Added asserts for shape testing. --- docs/tutorials/index.md| 2 + .../python/data_augmentation_with_masks.md | 248 + 2 files changed, 250 insertions(+) diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 8a597e9..c4d4690 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -119,6 +119,8 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of - [Simple autograd example](http://mxnet.incubator.apache.org/tutorials/gluon/autograd.html) +- [Data Augmentation with Masks (for Object Segmentation)](http://mxnet.incubator.apache.org/tutorials/python/data_augmentation_with_masks.html) + - [Inference using an ONNX model](http://mxnet.incubator.apache.org/tutorials/onnx/inference_on_onnx_model.html) diff --git a/docs/tutorials/python/data_augmentation_with_masks.md b/docs/tutorials/python/data_augmentation_with_masks.md new file mode 100644 index 000..28a57b6 --- /dev/null +++ b/docs/tutorials/python/data_augmentation_with_masks.md @@ -0,0 +1,248 @@ + +# Data Augmentation with Masks + +## Data Augmentation + +Data Augmentation is a regularization technique that's used to avoid overfitting when training Machine Learning models. Although the technique can be applied in a variety of domains, it's very common in Computer Vision, and this will be the focus of the tutorial. Adjustments are made to the original images in the training dataset before being used in training. Some example adjustments include translating, croping, scaling, rotating, changing brightness and contrast. We do this to reduce [...] + +https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/with_mask/orig_vs_aug.png; alt="Drawing" style="width: 480px;"/> + +## Masks + +Certain Computer Vision tasks (like [Object Segmentation](https://arxiv.org/abs/1506.06204)) require the use of 'masks', and we have to take extra care when using these in conjunction with data augmentation techniques. Given an underlying base image (with 3 channels), a masking channel can be added to provide additional metadata to certain regions of the base image. Masking channels often contain binary values, and these can be used to label a single class, e.g. to label a dog in the for [...] + +https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/with_mask/masks.png; alt="Drawing" style="width: 700px;"/> + +## Data Augmentation with Masks + +When we adjust the position of the base image as part of data augmentation, we also need to apply exactly the same operation to the associated masks. An example would be after applying a horizontal flip to the base image, we'd need to also flip the mask, to preserve the corresponsence between the base image and mask. + +Color changes to the base image don't need to be applied to the segmentation masks though; and may even lead to errors with the masks. An example with a RGB mask, would be accidentally converting a region of green for dog to blue for cat. + +https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/with_mask/aug_and_mask.png; alt="Drawing" style="width: 800px;"/> + +# Custom Dataset + +With Gluon it's easy to work with different types of data. You can write custom Datasets and plug them directly into a DataLoader which will handle batching. Segmentation tasks are structured in such a way that the data is the base image and the label is the mask, so we will create a custom Dataset for this. Our Dataset will return base images with their corresponsing masks. + +It will be based on the `mx.gluon.data.vision.ImageFolderDataset` for simplicity, and will load files from a single folder, containing images of the form `xyz.jpg` and their corresponsing mask `xyz_mask.png`. + +`__getitem__` must be implemented, as this will be used by the DataLoader. + + +```python +%matplotlib inline +import collections +import mxnet as mx # used version '1.0.0' at time of writing +from mxnet.gluon.data import dataset +import os +import numpy a
[incubator-mxnet] branch master updated: [MXNET-132] Change dead link in cpp-package (#10210)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new d07f59d [MXNET-132] Change dead link in cpp-package (#10210) d07f59d is described below commit d07f59d8b1aac4a9336d076626f3d8881926932b Author: ThomasDelteil <thomas.delte...@gmail.com> AuthorDate: Fri Mar 30 11:01:38 2018 -0700 [MXNET-132] Change dead link in cpp-package (#10210) * previous link is dead * Update README.md * Update README.md * Update README.md triggering build again, flaky test --- cpp-package/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cpp-package/README.md b/cpp-package/README.md index 93edd27..9dd3b01 100644 --- a/cpp-package/README.md +++ b/cpp-package/README.md @@ -1,6 +1,6 @@ # MXNet C++ Package -To build the package, please refer to <https://mxnet.incubator.apache.org/get_started/build_from_source.html#build-the-c-package>. +To build the C++ package, please refer to [this guide](<https://mxnet.incubator.apache.org/install/build_from_source#build-the-c-package>). A basic tutorial can be found at <https://mxnet.incubator.apache.org/tutorials/c++/basics.html>. -- To stop receiving notification emails like this one, please contact ind...@apache.org.
[incubator-mxnet] branch master updated: save profile outputs to different file for troubleshooting purposes (#9929)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new eaaf72d save profile outputs to different file for troubleshooting
purposes (#9929)
eaaf72d is described below
commit eaaf72d848794ef81a8fda394591730d3fc8845b
Author: Chris Olivier <cjolivie...@gmail.com>
AuthorDate: Wed Feb 28 16:10:16 2018 -0800
save profile outputs to different file for troubleshooting purposes (#9929)
---
tests/python/unittest/test_profiler.py | 24
1 file changed, 12 insertions(+), 12 deletions(-)
diff --git a/tests/python/unittest/test_profiler.py
b/tests/python/unittest/test_profiler.py
index b3a9a70..7654cd2 100644
--- a/tests/python/unittest/test_profiler.py
+++ b/tests/python/unittest/test_profiler.py
@@ -22,8 +22,7 @@ from mxnet import profiler
import time
import os
-def enable_profiler(run=True, continuous_dump=False, aggregate_stats=False):
-profile_filename = 'test_profile.json'
+def enable_profiler(profile_filename, run=True, continuous_dump=False,
aggregate_stats=False):
profiler.set_config(profile_symbolic=True,
profile_imperative=True,
profile_memory=True,
@@ -42,7 +41,7 @@ def test_profiler():
begin_profiling_iter = 2
end_profiling_iter = 4
-enable_profiler(False, False)
+enable_profiler('test_profiler.json', False, False)
A = mx.sym.Variable('A')
B = mx.sym.Variable('B')
@@ -77,14 +76,14 @@ def test_profiler():
def test_profile_create_domain():
-enable_profiler()
+enable_profiler('test_profile_create_domain.json')
domain = profiler.Domain(name='PythonDomain')
print("Domain created: {}".format(str(domain)))
profiler.set_state('stop')
def test_profile_create_domain_dept():
-profiler.profiler_set_config(mode='symbolic', filename='temp.json')
+profiler.profiler_set_config(mode='symbolic',
filename='test_profile_create_domain_dept.json')
profiler.set_state('run')
domain = profiler.Domain(name='PythonDomain')
print("Domain created: {}".format(str(domain)))
@@ -103,7 +102,7 @@ def test_profile_task():
logging.info(template.format(*objects))
logging.basicConfig()
-enable_profiler()
+enable_profiler('test_profile_task.json')
python_domain = profiler.Domain('PythonDomain::test_profile_task')
task = profiler.Task(python_domain, "test_profile_task")
task.start()
@@ -129,7 +128,7 @@ def test_profile_frame():
logging.info(template.format(*objects))
logging.basicConfig()
-enable_profiler()
+enable_profiler('test_profile_frame.json')
python_domain = profiler.Domain('PythonDomain::test_profile_frame')
frame = profiler.Frame(python_domain, "test_profile_frame")
frame.start()
@@ -156,7 +155,7 @@ def test_profile_event(do_enable_profiler=True):
logging.basicConfig()
if do_enable_profiler is True:
- enable_profiler()
+enable_profiler('test_profile_event.json')
event = profiler.Event("test_profile_event")
event.start()
start = time.time()
@@ -171,7 +170,7 @@ def test_profile_event(do_enable_profiler=True):
def test_profile_tune_pause_resume():
-enable_profiler()
+enable_profiler('test_profile_tune_pause_resume.json')
profiler.pause()
# "test_profile_task" should *not* show up in tuning analysis
test_profile_task()
@@ -199,7 +198,7 @@ def test_profile_counter(do_enable_profiler=True):
logging.info(template.format(*objects))
if do_enable_profiler is True:
- enable_profiler()
+enable_profiler('test_profile_counter.json')
python_domain = profiler.Domain('PythonDomain::test_profile_counter')
counter = profiler.Counter(python_domain,
"PythonCounter::test_profile_counter")
counter.set_value(5)
@@ -213,7 +212,8 @@ def test_profile_counter(do_enable_profiler=True):
def test_continuous_profile_and_instant_marker():
-enable_profiler(True, True, True)
+file_name = 'test_continuous_profile_and_instant_marker.json'
+enable_profiler(file_name, True, True, True)
python_domain = profiler.Domain('PythonDomain::test_continuous_profile')
last_file_size = 0
for i in range(5):
@@ -223,7 +223,7 @@ def test_continuous_profile_and_instant_marker():
test_profile_counter(False)
profiler.dump(False)
# File size should keep increasing
-new_file_size = os.path.getsize("test_profile.json")
+new_file_size = os.path.getsize(file_name)
assert new_file_size >= last_file_size
last_file_size = new_file_size
profiler.dump(False)
--
To stop receiving notification emails like this one, please contact
ind...@apache.org.
[incubator-mxnet] branch master updated: Some more license adds (#9504)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new cb52708 Some more license adds (#9504) cb52708 is described below commit cb5270899722b0c44db0d871ded54b11bd28fcdd Author: mbaijal <30911248+mbai...@users.noreply.github.com> AuthorDate: Mon Jan 22 13:13:04 2018 -0800 Some more license adds (#9504) --- tests/ci_build/with_the_same_user | 17 + tests/jenkins/format| 17 + tools/caffe_translator/gradle.properties| 17 + .../gradle/wrapper/gradle-wrapper.properties| 17 + tools/caffe_translator/settings.gradle | 17 + 5 files changed, 85 insertions(+) diff --git a/tests/ci_build/with_the_same_user b/tests/ci_build/with_the_same_user index 9220156..19f0de1 100755 --- a/tests/ci_build/with_the_same_user +++ b/tests/ci_build/with_the_same_user @@ -1,5 +1,22 @@ #!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + # This script is a wrapper creating the same user inside container as the one # running the ci_build.sh outside the container. It also set the home directory # for the user inside container to match the same absolute path as the workspace diff --git a/tests/jenkins/format b/tests/jenkins/format index 93bdbef..405f024 100644 --- a/tests/jenkins/format +++ b/tests/jenkins/format @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + # match line starting with 'error ', case-insensitive error /(?i)^error / diff --git a/tools/caffe_translator/gradle.properties b/tools/caffe_translator/gradle.properties index f115eaa..7e5363b 100644 --- a/tools/caffe_translator/gradle.properties +++ b/tools/caffe_translator/gradle.properties @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + antlrVersion=4.7 signing.keyId= diff --git a/tools/caffe_translator/gradle/wrapper/gradle-wrapper.properties b/tools/caffe_translator/gradle/wrapper/gradle-wrapper.properties index 0e680f3..a4daf1d 100644 --- a/tools/caffe_translator/gradle/wrapper/gradle-wrapper.properties +++ b/tools/caffe_translator/gradle/wrapper/gradle-wrapper.properties @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this fil
[incubator-mxnet] branch v1.0.0 updated: Reorganize the tutorials index page (#9033)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch v1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/v1.0.0 by this push: new ec0a65a Reorganize the tutorials index page (#9033) ec0a65a is described below commit ec0a65a6f79c1c46b2ef09412731901a56b8707f Author: Indhu Bharathi <indhubhara...@gmail.com> AuthorDate: Wed Dec 13 21:01:44 2017 -0800 Reorganize the tutorials index page (#9033) * Minor changes to tutorials index page. * Reorganize the tutorials index page. --- docs/tutorials/index.md | 64 - 1 file changed, 42 insertions(+), 22 deletions(-) diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index d20a821..50f8578 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -1,16 +1,27 @@ # Tutorials +## NDArray + +NDArray is MXNet’s primary tool for storing and transforming data. NDArrays are similar to NumPy's multi-dimensional array. However, they confer a few key advantages. First, NDArrays support asynchronous computation on CPU, GPU, and distributed cloud architectures. Second, they provide support for automatic differentiation. These properties make NDArray an ideal library for machine learning, both for researchers and engineers launching production systems. + +- [Manipulate data the MXNet way with ndarray](http://gluon.mxnet.io/chapter01_crashcourse/ndarray.html) + + +## Automatic gradients + +MXNet makes it easier to calculate derivatives by automatically calculating them while writing ordinary imperative code. Every time you a make pass through your model, autograd builds a graph on the fly, through which it can immediately backpropagate gradients. + +- [Automatic differentiation with autograd](http://gluon.mxnet.io/chapter01_crashcourse/autograd.html) + + ## Gluon -Gluon is the high-level interface for MXNet. It is more intuitive and easier to use than the lower level interface. -Gluon supports dynamic (define-by-run) graphs with JIT-compilation to achieve both flexibility and efficiency. +Gluon is MXNet's imperative API. It is more intuitive and easier to use than the symbolic API. Gluon supports dynamic (define-by-run) graphs with JIT-compilation to achieve both flexibility and efficiency. -This is a selected subset of Gluon tutorials that explains basic usage of Gluon and fundamental concepts in deep learning. For the comprehensive tutorial on Gluon that covers topics from basic statistics and probability theory to reinforcement learning and recommender systems, please see [gluon.mxnet.io](http://gluon.mxnet.io). +This is a selected subset of Gluon tutorials that explains basic usage of Gluon and fundamental concepts in deep learning. For the comprehensive tutorial on Gluon that covers topics from basic statistics and probability theory to reinforcement learning and recommender systems, please see gluon.mxnet.io. ### Basics -- [Manipulate data the MXNet way with ndarray](http://gluon.mxnet.io/chapter01_crashcourse/ndarray.html) -- [Automatic differentiation with autograd](http://gluon.mxnet.io/chapter01_crashcourse/autograd.html) - [Linear regression with gluon](http://gluon.mxnet.io/chapter02_supervised-learning/linear-regression-gluon.html) - [Serialization - saving, loading and checkpointing](http://gluon.mxnet.io/chapter03_deep-neural-networks/serialization.html) @@ -24,52 +35,61 @@ This is a selected subset of Gluon tutorials that explains basic usage of Gluon - [Plumbing: A look under the hood of gluon](http://gluon.mxnet.io/chapter03_deep-neural-networks/plumbing.html) - [Designing a custom layer with gluon](http://gluon.mxnet.io/chapter03_deep-neural-networks/custom-layer.html) -- [Fast, portable neural networks with Gluon HybridBlocks](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html) - [Training on multiple GPUs with gluon](http://gluon.mxnet.io/chapter07_distributed-learning/multiple-gpus-gluon.html) -## MXNet -These tutorials introduce a few fundamental concepts in deep learning and how to implement them in _MXNet_. The _Basics_ section contains tutorials on manipulating arrays, building networks, loading/preprocessing data, etc. The _Training and Inference_ section talks about implementing Linear Regression, training a Handwritten digit classifier using MLP and CNN, running inferences using a pre-trained model, and lastly, efficiently training a large scale image classifier. +## Symbolic Interface + +MXNet's symbolic interface lets users define a computation graph first and then execute it using MXNet. This enables MXNet to perform a lot of optimizations that are not possible in imperative execution (like operator folding and safe reuse of memory used by temporary variables). -### Basics ```eval_rst .. toctree:: :maxdepth: 1 - basic/n
[incubator-mxnet] branch master updated: Minor changes to Caffe Translator (#8939)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 9d4bb9c Minor changes to Caffe Translator (#8939) 9d4bb9c is described below commit 9d4bb9c53c00d3b8f8ddd1e9e92ac8cbeb885111 Author: Indhu Bharathi <indhubhara...@gmail.com> AuthorDate: Fri Dec 8 16:28:17 2017 -0800 Minor changes to Caffe Translator (#8939) * - Add license to string template files. - Add license to gradlew - Some bug fixes and refactoring for optimizer generation. - Language change in comment that goes into generated code. - Don't generate CaffeLoss layer for Accuracy layer. It is now being translated to MXNet Accuracy metrics. - Minor bug fix in searching for the correct optimizer template. - Bump verion up to 0.9.2 * Add license for Optimizer.java * Code cleanup. --- tools/caffe_translator/build.gradle| 2 +- tools/caffe_translator/gradlew | 17 + .../java/io/mxnet/caffetranslator/Converter.java | 75 +++--- .../java/io/mxnet/caffetranslator/Optimizer.java | 48 ++ .../main/java/io/mxnet/caffetranslator/Solver.java | 52 ++- .../src/main/resources/templates/accuracy.st | 18 ++ .../src/main/resources/templates/activation.st | 18 ++ .../src/main/resources/templates/add.st| 18 ++ .../src/main/resources/templates/batchnorm.st | 18 ++ .../src/main/resources/templates/concat.st | 18 ++ .../src/main/resources/templates/convolution.st| 18 ++ .../src/main/resources/templates/deconvolution.st | 18 ++ .../src/main/resources/templates/dropout.st| 18 ++ .../src/main/resources/templates/fc.st | 18 ++ .../src/main/resources/templates/flatten.st| 18 ++ .../src/main/resources/templates/group.st | 18 ++ .../src/main/resources/templates/imports.st| 18 ++ .../src/main/resources/templates/init_params.st| 18 ++ .../src/main/resources/templates/iterator.st | 18 ++ .../src/main/resources/templates/logging.st| 18 ++ .../src/main/resources/templates/lrn.st| 18 ++ .../src/main/resources/templates/lrpolicy_exp.st | 18 ++ .../src/main/resources/templates/lrpolicy_inv.st | 18 ++ .../main/resources/templates/lrpolicy_multistep.st | 18 ++ .../src/main/resources/templates/lrpolicy_poly.st | 18 ++ .../main/resources/templates/lrpolicy_sigmoid.st | 18 ++ .../src/main/resources/templates/lrpolicy_step.st | 18 ++ .../src/main/resources/templates/maxium.st | 18 ++ .../main/resources/templates/metrics_classes.st| 27 ++-- .../src/main/resources/templates/mul.st| 18 ++ .../src/main/resources/templates/opt_adadelta.st | 32 + .../src/main/resources/templates/opt_adagrad.st| 28 .../src/main/resources/templates/opt_adam.st | 36 +++ .../src/main/resources/templates/opt_default.st| 15 - .../src/main/resources/templates/opt_nesterov.st | 28 .../src/main/resources/templates/opt_rmsprop.st| 32 + .../src/main/resources/templates/opt_sgd.st| 36 --- .../src/main/resources/templates/opt_vars.st | 24 +++ .../main/resources/templates/param_initializer.st | 18 ++ .../src/main/resources/templates/params_loader.st | 18 ++ .../src/main/resources/templates/permute.st| 18 ++ .../src/main/resources/templates/pooling.st| 18 ++ .../src/main/resources/templates/power.st | 18 ++ .../src/main/resources/templates/runner.st | 18 ++ .../src/main/resources/templates/softmaxoutput.st | 18 ++ .../src/main/resources/templates/symbols.stg | 18 ++ .../src/main/resources/templates/top_k_accuracy.st | 18 ++ .../src/main/resources/templates/var.st| 18 ++ 48 files changed, 979 insertions(+), 85 deletions(-) diff --git a/tools/caffe_translator/build.gradle b/tools/caffe_translator/build.gradle index 4206767..da5e900 100644 --- a/tools/caffe_translator/build.gradle +++ b/tools/caffe_translator/build.gradle @@ -10,7 +10,7 @@ apply plugin: 'maven' apply plugin: 'signing' group 'org.caffetranslator' -version '0.9.1' +version '0.9.2' def isReleaseBuild def repositoryUrl diff --git a/tools/caffe_translator/gradlew b/tools/caffe_translator/gradlew index cccdd3d..07cc915 100755 --- a/tools/caffe_translator/gradlew +++ b/tools/caffe_translator/gradlew @@ -1,5 +1,22 @@ #!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright own
[incubator-mxnet] branch master updated: update ps lite (#8327)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 349ebc3 update ps lite (#8327) 349ebc3 is described below commit 349ebc36ff3693376287098d280b11069dafbb55 Author: Eric Junyuan Xie <piiswr...@users.noreply.github.com> AuthorDate: Wed Oct 18 01:49:04 2017 -0700 update ps lite (#8327) --- ps-lite | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ps-lite b/ps-lite index acdb698..bdd4c67 16 --- a/ps-lite +++ b/ps-lite @@ -1 +1 @@ -Subproject commit acdb698fa3bb80929ef83bb37c705f025e119b82 +Subproject commit bdd4c67e9e34dc0b8350ce306b0caa737eb31c83 -- To stop receiving notification emails like this one, please contact ['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch master updated: fixed broken links. https was pointing to http for mxnet.io (#8300)
This is an automated email from the ASF dual-hosted git repository. indhub pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git The following commit(s) were added to refs/heads/master by this push: new 9034855 fixed broken links. https was pointing to http for mxnet.io (#8300) 9034855 is described below commit 9034855a1f251310ee487ff780f69912d5a844c3 Author: thinksanky <31976455+thinksa...@users.noreply.github.com> AuthorDate: Tue Oct 17 15:27:25 2017 -0700 fixed broken links. https was pointing to http for mxnet.io (#8300) --- docs/tutorials/r/symbol.md | 2 +- docs/tutorials/sparse/row_sparse.md | 2 +- docs/tutorials/sparse/train.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tutorials/r/symbol.md b/docs/tutorials/r/symbol.md index 63f3a53..6ab4dc2 100644 --- a/docs/tutorials/r/symbol.md +++ b/docs/tutorials/r/symbol.md @@ -104,7 +104,7 @@ In the example, *net* is used as a function to apply to an existing symbol ## Training a Neural Net -The [model API](../../../R-package/R/model.R) is a thin wrapper around the symbolic executors to support neural net training. +The [model API](https://github.com/apache/incubator-mxnet/blob/master/R-package/R/model.R) is a thin wrapper around the symbolic executors to support neural net training. We encourage you to read [Symbolic Configuration and Execution in Pictures for python package](../../api/python/symbol_in_pictures/symbol_in_pictures.md)for a detailed explanation of concepts in pictures. diff --git a/docs/tutorials/sparse/row_sparse.md b/docs/tutorials/sparse/row_sparse.md index e2f0a12..6a69341 100644 --- a/docs/tutorials/sparse/row_sparse.md +++ b/docs/tutorials/sparse/row_sparse.md @@ -271,7 +271,7 @@ rsp_retained = mx.nd.sparse.retain(rsp, mx.nd.array([0, 1])) ## Sparse Operators and Storage Type Inference -Operators that have specialized implementation for sparse arrays can be accessed in ``mx.nd.sparse``. You can read the [mxnet.ndarray.sparse API documentation](https://mxnet.io/versions/master/api/python/ndarray/sparse.html) to find what sparse operators are available. +Operators that have specialized implementation for sparse arrays can be accessed in ``mx.nd.sparse``. You can read the [mxnet.ndarray.sparse API documentation](http://mxnet.io/versions/master/api/python/ndarray/sparse.html) to find what sparse operators are available. ```python diff --git a/docs/tutorials/sparse/train.md b/docs/tutorials/sparse/train.md index d6e3f4e..22ce039 100644 --- a/docs/tutorials/sparse/train.md +++ b/docs/tutorials/sparse/train.md @@ -99,7 +99,7 @@ f = mx.sym.sparse.elemwise_add(c, c) ### Storage Type Inference What will be the output storage types of sparse symbols? In MXNet, for any sparse symbol, the result storage types are inferred based on storage types of inputs. -You can read the [Sparse Symbol API](https://mxnet.io/versions/master/api/python/symbol/sparse.html) documentation to find what output storage types are. In the example below we will try out the storage types introduced in the Row Sparse and Compressed Sparse Row tutorials: `default` (dense), `csr`, and `row_sparse`. +You can read the [Sparse Symbol API](http://mxnet.io/versions/master/api/python/symbol/sparse.html) documentation to find what output storage types are. In the example below we will try out the storage types introduced in the Row Sparse and Compressed Sparse Row tutorials: `default` (dense), `csr`, and `row_sparse`. ```python -- To stop receiving notification emails like this one, please contact ['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch v0.12.0 updated: Disable Caffe Converter integration tests. The test is failing because it is not able to download files from caffe server. (#8284)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch v0.12.0
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/v0.12.0 by this push:
new 87e4bb9 Disable Caffe Converter integration tests. The test is
failing because it is not able to download files from caffe server. (#8284)
87e4bb9 is described below
commit 87e4bb9f91d27ca98eeb2c79c187af823bd5e4ea
Author: Indhu Bharathi <indhubhara...@gmail.com>
AuthorDate: Sun Oct 15 09:33:06 2017 -0700
Disable Caffe Converter integration tests. The test is failing because it
is not able to download files from caffe server. (#8284)
This test can be enabled again after
https://github.com/apache/incubator-mxnet/issues/8283 is fixed.
---
Jenkinsfile | 11 ---
1 file changed, 11 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index e48e552..a6f23e4 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -485,17 +485,6 @@ try {
}
}
},
- 'Caffe': {
-node('mxnetlinux') {
- ws('workspace/it-caffe') {
-init_git()
-unpack_lib('gpu')
-timeout(time: max_time, unit: 'MINUTES') {
- sh "${docker_run} caffe_gpu PYTHONPATH=/caffe/python:./python
python tools/caffe_converter/test_converter.py"
-}
- }
-}
- },
'cpp-package': {
node('mxnetlinux') {
ws('workspace/it-cpp-package') {
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch master updated: Temporarily disable 'test_row_sparse_pull' on GPU. (#8265)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new efef7b7 Temporarily disable 'test_row_sparse_pull' on GPU. (#8265)
efef7b7 is described below
commit efef7b7b4e24584e059a9c0ce995f02cc262cc08
Author: Indhu Bharathi <indhubhara...@gmail.com>
AuthorDate: Sat Oct 14 19:44:32 2017 -0700
Temporarily disable 'test_row_sparse_pull' on GPU. (#8265)
* Temporarily disable 'test_row_sparse_pull' on GPU. Can be enabled back
after https://github.com/apache/incubator-mxnet/issues/8262 is resolved.
* Disable test_row_sparse_pull for GPU (not CPU)
* Fix build
---
tests/python/gpu/test_kvstore_gpu.py| 2 +
tests/python/gpu/test_operator_gpu.py | 19 +-
tests/python/unittest/test_kvstore.py | 1 +
tests/python/unittest/test_optimizer.py | 67 +
4 files changed, 47 insertions(+), 42 deletions(-)
diff --git a/tests/python/gpu/test_kvstore_gpu.py
b/tests/python/gpu/test_kvstore_gpu.py
index ffc0cc1..517d2e7 100644
--- a/tests/python/gpu/test_kvstore_gpu.py
+++ b/tests/python/gpu/test_kvstore_gpu.py
@@ -18,6 +18,7 @@
# pylint: skip-file
import mxnet as mx
import numpy as np
+import unittest
from mxnet.test_utils import assert_almost_equal, default_context
shape = (4, 4)
@@ -35,6 +36,7 @@ def init_kv_with_str(stype='default'):
return kv
+@unittest.skip("Test fails intermittently. Temporarily disabled until fixed.
Tracked at https://github.com/apache/incubator-mxnet/issues/8262;)
def test_row_sparse_pull():
kv = init_kv_with_str('row_sparse')
kv.init('e', mx.nd.ones(shape).tostype('row_sparse'))
diff --git a/tests/python/gpu/test_operator_gpu.py
b/tests/python/gpu/test_operator_gpu.py
index 2f2c3a8..b1f43f3 100644
--- a/tests/python/gpu/test_operator_gpu.py
+++ b/tests/python/gpu/test_operator_gpu.py
@@ -21,6 +21,7 @@ import time
import unittest
import mxnet as mx
import numpy as np
+import unittest
from mxnet.test_utils import check_consistency, set_default_context,
assert_almost_equal
from numpy.testing import assert_allclose
@@ -1358,16 +1359,16 @@ def test_rnn_layer():
def test_sequence_reverse():
check_sequence_reverse(mx.gpu(0))
+@unittest.skip("Test fails intermittently. Temporarily disabled until fixed.
Tracked at https://github.com/apache/incubator-mxnet/issues/8211;)
+def test_autograd_save_memory():
+x = mx.nd.zeros((128, 512, 512), ctx=mx.gpu(0))
+x.attach_grad()
-#def test_autograd_save_memory():
-#x = mx.nd.zeros((128, 512, 512), ctx=mx.gpu(0))
-#x.attach_grad()
-#
-#with mx.autograd.record():
-#for i in range(200):
-#x = x + 1
-#x.wait_to_read()
-#x.backward()
+with mx.autograd.record():
+for i in range(200):
+x = x + 1
+x.wait_to_read()
+x.backward()
def test_gluon_ctc_consistency():
loss = mx.gluon.loss.CTCLoss()
diff --git a/tests/python/unittest/test_kvstore.py
b/tests/python/unittest/test_kvstore.py
index 37d44e0..fc9e3be 100644
--- a/tests/python/unittest/test_kvstore.py
+++ b/tests/python/unittest/test_kvstore.py
@@ -18,6 +18,7 @@
# pylint: skip-file
import mxnet as mx
import numpy as np
+import unittest
from mxnet.test_utils import rand_ndarray, assert_almost_equal,
assert_exception
from mxnet.base import py_str, MXNetError
diff --git a/tests/python/unittest/test_optimizer.py
b/tests/python/unittest/test_optimizer.py
index 62a1d14..8666b9e 100644
--- a/tests/python/unittest/test_optimizer.py
+++ b/tests/python/unittest/test_optimizer.py
@@ -18,6 +18,7 @@
import numpy as np
import mxnet as mx
import mxnet.lr_scheduler as lr_scheduler
+import unittest
from nose.tools import raises
import math
from mxnet.test_utils import *
@@ -532,39 +533,39 @@ class PyRMSProp(mx.optimizer.Optimizer):
if self.clip_weights:
mx.ndarray.clip(weight, -self.clip_weights, self.clip_weights,
out=weight)
-#def test_rms():
-#mx.random.seed(0)
-#opt1 = PyRMSProp
-#opt2 = mx.optimizer.RMSProp
-#shape = (3, 4, 5)
-#cg_options = [{}, {'clip_gradient': 0.4}, {'clip_gradient': 0.5}]
-#cw_options = [{}, {'clip_weights': 0.01}]
-#center_options = [{}, {'centered': False}, {'centered': True}]
-#rg_options = [{}, {'rescale_grad': 0.14}, {'rescale_grad': 0.8}]
-#wd_options = [{}, {'wd': 0.03}, {'wd': 0.05}, {'wd': 0.07}]
-#mp_options = [{}, {'multi_precision': False}, {'multi_precision': True}]
-#for dtype in [np.float16, np.float32]:
-#for cw_option in cw_options:
-#for cg_option in cg_options:
-#for center_option in center_options:
-#for rg_option in rg_options:
-#for wd_option in wd_options:
-#
[incubator-mxnet] branch ci-test updated: Fix typo.
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch ci-test
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/ci-test by this push:
new f20d45b Fix typo.
f20d45b is described below
commit f20d45be3a46849308f70390a62ecda98461b194
Author: Indu Bharathi <indhubhara...@gmail.com>
AuthorDate: Sat Oct 14 16:15:07 2017 +
Fix typo.
---
Jenkinsfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 9f5e33d..e1c5933 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -411,7 +411,7 @@ try {
init_git()
unpack_lib('cpu')
withEnv(['MXNET_HOME=/workspace/ut-julia-cpu']) {
- timeout(time: max_timem unit: 'MINUTES') {
+ timeout(time: max_time, unit: 'MINUTES') {
sh "${docker_run} cpu julia -e 'Pkg.clone(\"julia-package\",
\"MXNet\")'"
sh "${docker_run} cpu julia -e 'Pkg.build(\"MXNet\")'"
sh "${docker_run} cpu julia -e 'Pkg.test(\"MXNet\")'"
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch ci-test updated: WIP: Julia CI build (#8175)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch ci-test
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/ci-test by this push:
new 1b234fb WIP: Julia CI build (#8175)
1b234fb is described below
commit 1b234fb0b97780a109087f69ac757d78361efde7
Author: Iblis Lin <ib...@hs.ntnu.edu.tw>
AuthorDate: Fri Oct 13 19:17:01 2017 +0800
WIP: Julia CI build (#8175)
* Delete workspace before build. (#8195)
* Delete workspace before build.
* Move 'deleteDir' outside the retry.
* Jenkins: fix Jenkinsfile indentation
* add Julia package as a submodule and enable CPU build
---
.gitmodules | 3 ++
Jenkinsfile | 69 -
julia-package | 1 +
tests/ci_build/Dockerfile.cpu | 2 +
tests/ci_build/install/install_julia.sh | 10 ++---
tools/license_header.py | 1 +
6 files changed, 54 insertions(+), 32 deletions(-)
diff --git a/.gitmodules b/.gitmodules
index 7a76cba..eb22b2f 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -16,3 +16,6 @@
[submodule "cub"]
path = cub
url = https://github.com/dmlc/cub
+[submodule "julia-package"]
+ path = julia-package
+ url = https://github.com/dmlc/MXNet.jl.git
diff --git a/Jenkinsfile b/Jenkinsfile
index edea99e..9f5e33d 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -19,7 +19,7 @@ def init_git() {
timeout(time: 2, unit: 'MINUTES') {
checkout scm
sh 'git submodule update --init'
-sh 'git clean -d -f'
+sh 'git clean -d -f'
}
} catch (exc) {
error "Failed to fetch source codes"
@@ -35,7 +35,7 @@ def init_git_win() {
timeout(time: 2, unit: 'MINUTES') {
checkout scm
bat 'git submodule update --init'
-bat 'git clean -d -f'
+bat 'git clean -d -f'
}
} catch (exc) {
error "Failed to fetch source codes"
@@ -405,7 +405,22 @@ try {
}
}
},
- 'Python 2: CPU Win':{
+ 'Julia 0.6: CPU': {
+node('mxnetlinux') {
+ ws('workspace/ut-julia-cpu') {
+init_git()
+unpack_lib('cpu')
+withEnv(['MXNET_HOME=/workspace/ut-julia-cpu']) {
+ timeout(time: max_timem unit: 'MINUTES') {
+sh "${docker_run} cpu julia -e 'Pkg.clone(\"julia-package\",
\"MXNet\")'"
+sh "${docker_run} cpu julia -e 'Pkg.build(\"MXNet\")'"
+sh "${docker_run} cpu julia -e 'Pkg.test(\"MXNet\")'"
+ }
+}
+ }
+}
+ },
+ 'Python 2: CPU Win': {
node('mxnetwindows') {
ws('workspace/ut-python-cpu') {
init_git_win()
@@ -419,32 +434,32 @@ try {
del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc
C:\\mxnet\\test_cpu.bat"""
}
- }
- },
- 'Python 3: CPU Win': {
- node('mxnetwindows') {
+}
+ },
+ 'Python 3: CPU Win': {
+node('mxnetwindows') {
ws('workspace/ut-python-cpu') {
init_git_win()
unstash 'vc14_cpu'
bat '''rmdir /s/q pkg_vc14_cpu
7z x -y vc14_cpu.7z'''
- bat """xcopy C:\\mxnet\\data data /E /I /Y
+bat """xcopy C:\\mxnet\\data data /E /I /Y
xcopy C:\\mxnet\\model model /E /I /Y
call activate py3
set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_cpu\\python
del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc
C:\\mxnet\\test_cpu.bat"""
}
- }
- },
- 'Python 2: GPU Win':{
- node('mxnetwindows') {
- ws('workspace/ut-python-gpu') {
- init_git_win()
- unstash 'vc14_gpu'
- bat '''rmdir /s/q pkg_vc14_gpu
+}
+ },
+ 'Python 2: GPU Win': {
+node('mxnetwindows') {
+ ws('workspace/ut-python-gpu') {
+init_git_win()
+unstash 'vc14_gpu'
+bat '''rmdir /s/q pkg_vc14_gpu
7z x -y vc14_gpu.7z'''
- bat """xcopy C:\\mxnet\\data data /E /I /Y
+bat """xcopy C:\\mxnet\\data data /E /I /Y
xcopy C:\\mxnet\\model model /E /I /Y
call activate py2
set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_gpu\\python
@@ -452,23 +467,23 @@ try {
C:\\mxnet\\test_gpu.bat"""
}
}
- },
- 'Python 3: GPU Win':{
- node('mxnetwindows') {
- ws('workspace/ut-python-gpu') {
- init_git_win()
- unstash 'vc14_gpu'
- bat '''rmdir /s
[incubator-mxnet] branch master updated: Temporarily disable some unit tests to fix the build (#8253)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new a05ac49 Temporarily disable some unit tests to fix the build (#8253)
a05ac49 is described below
commit a05ac492379f479769b948b2268a175233e96784
Author: Indhu Bharathi <indhubhara...@gmail.com>
AuthorDate: Fri Oct 13 03:40:19 2017 -0700
Temporarily disable some unit tests to fix the build (#8253)
* Temporarily disable the following unit tests that have been causing build
failures:
test_rms:
This can be re-enabled once
https://github.com/apache/incubator-mxnet/issues/8230 is fixed.
test_autograd_save_memory:
This can be re-enabled once
https://github.com/apache/incubator-mxnet/issues/8211 is fixed.
---
tests/python/gpu/test_operator_gpu.py | 18 -
tests/python/unittest/test_optimizer.py | 65 +
2 files changed, 42 insertions(+), 41 deletions(-)
diff --git a/tests/python/gpu/test_operator_gpu.py
b/tests/python/gpu/test_operator_gpu.py
index c991e50..2f2c3a8 100644
--- a/tests/python/gpu/test_operator_gpu.py
+++ b/tests/python/gpu/test_operator_gpu.py
@@ -1359,15 +1359,15 @@ def test_sequence_reverse():
check_sequence_reverse(mx.gpu(0))
-def test_autograd_save_memory():
-x = mx.nd.zeros((128, 512, 512), ctx=mx.gpu(0))
-x.attach_grad()
-
-with mx.autograd.record():
-for i in range(200):
-x = x + 1
-x.wait_to_read()
-x.backward()
+#def test_autograd_save_memory():
+#x = mx.nd.zeros((128, 512, 512), ctx=mx.gpu(0))
+#x.attach_grad()
+#
+#with mx.autograd.record():
+#for i in range(200):
+#x = x + 1
+#x.wait_to_read()
+#x.backward()
def test_gluon_ctc_consistency():
loss = mx.gluon.loss.CTCLoss()
diff --git a/tests/python/unittest/test_optimizer.py
b/tests/python/unittest/test_optimizer.py
index 1a56859..62a1d14 100644
--- a/tests/python/unittest/test_optimizer.py
+++ b/tests/python/unittest/test_optimizer.py
@@ -532,38 +532,39 @@ class PyRMSProp(mx.optimizer.Optimizer):
if self.clip_weights:
mx.ndarray.clip(weight, -self.clip_weights, self.clip_weights,
out=weight)
-def test_rms():
-mx.random.seed(0)
-opt1 = PyRMSProp
-opt2 = mx.optimizer.RMSProp
-shape = (3, 4, 5)
-cg_options = [{}, {'clip_gradient': 0.4}, {'clip_gradient': 0.5}]
-cw_options = [{}, {'clip_weights': 0.01}]
-center_options = [{}, {'centered': False}, {'centered': True}]
-rg_options = [{}, {'rescale_grad': 0.14}, {'rescale_grad': 0.8}]
-wd_options = [{}, {'wd': 0.03}, {'wd': 0.05}, {'wd': 0.07}]
-mp_options = [{}, {'multi_precision': False}, {'multi_precision': True}]
-for dtype in [np.float16, np.float32]:
-for cw_option in cw_options:
-for cg_option in cg_options:
-for center_option in center_options:
-for rg_option in rg_options:
-for wd_option in wd_options:
-for mp_option in mp_options:
-kwarg = {}
-kwarg.update(cw_option)
-kwarg.update(cg_option)
-kwarg.update(center_option)
-kwarg.update(rg_option)
-kwarg.update(wd_option)
-kwarg.update(mp_option)
-if (dtype == np.float16 and
-('multi_precision' not in kwarg or
-not kwarg['multi_precision'])):
-continue
-compare_optimizer(opt1(**kwarg),
opt2(**kwarg), shape, dtype)
-if (default_context() == mx.cpu()):
-compare_optimizer(opt1(**kwarg),
opt2(**kwarg), shape, dtype, g_stype='row_sparse')
+#def test_rms():
+#mx.random.seed(0)
+#opt1 = PyRMSProp
+#opt2 = mx.optimizer.RMSProp
+#shape = (3, 4, 5)
+#cg_options = [{}, {'clip_gradient': 0.4}, {'clip_gradient': 0.5}]
+#cw_options = [{}, {'clip_weights': 0.01}]
+#center_options = [{}, {'centered': False}, {'centered': True}]
+#rg_options = [{}, {'rescale_grad': 0.14}, {'rescale_grad': 0.8}]
+#wd_options = [{}, {'wd': 0.03}, {'wd': 0.05}, {'wd': 0.07}]
+#mp_options = [{}, {'multi_precision': False}, {'multi_precision': True}]
+#for dtype in [np.float16, np.float32]:
+#for cw_option in cw_options:
+#for cg_option in cg_options:
+#for center_option in center_options:
+#for rg_option in rg_options:
+#for wd_option in wd_o
[incubator-mxnet] branch master updated: Fixing the Caught error (#8199)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 0092fa2 Fixing the Caught error (#8199)
0092fa2 is described below
commit 0092fa26e02247164fec11bf86baee8d38fc9a83
Author: Gautam Kumar <gaku...@fb.com>
AuthorDate: Thu Oct 12 18:10:30 2017 -0700
Fixing the Caught error (#8199)
---
Jenkinsfile | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 14d1ce4..e48e552 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -23,7 +23,7 @@ def init_git() {
}
} catch (exc) {
deleteDir()
- error "Failed to fetch source codes"
+ error "Failed to fetch source codes with ${exc}"
sleep 2
}
}
@@ -40,7 +40,7 @@ def init_git_win() {
}
} catch (exc) {
deleteDir()
- error "Failed to fetch source codes"
+ error "Failed to fetch source codes with ${exc}"
sleep 2
}
}
@@ -54,7 +54,7 @@ def make(docker_type, make_flag) {
try {
sh "${docker_run} ${docker_type} make ${make_flag}"
} catch (exc) {
- echo 'Incremental compilation failed. Fall back to build from scratch'
+ echo 'Incremental compilation failed with ${exc}. Fall back to build
from scratch'
sh "${docker_run} ${docker_type} sudo make clean"
sh "${docker_run} ${docker_type} sudo make -C amalgamation/ clean"
sh "${docker_run} ${docker_type} make ${make_flag}"
@@ -525,7 +525,7 @@ try {
currentBuild.result = "SUCCESS"
} catch (caughtError) {
node("mxnetlinux") {
-sh "echo caught error"
+sh "echo caught ${caughtError}"
err = caughtError
currentBuild.result = "FAILURE"
}
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch master updated: increase number of training epochs. (#8214)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new c1beac5 increase number of training epochs. (#8214)
c1beac5 is described below
commit c1beac566f898a122d53a19fef32d69497180251
Author: Sergey Kolychev <sergeykolychev.git...@gmail.com>
AuthorDate: Wed Oct 11 14:50:53 2017 -0700
increase number of training epochs. (#8214)
---
perl-package/AI-MXNet/t/test_loss.t | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/perl-package/AI-MXNet/t/test_loss.t
b/perl-package/AI-MXNet/t/test_loss.t
index bfdf304..f98d4f2 100644
--- a/perl-package/AI-MXNet/t/test_loss.t
+++ b/perl-package/AI-MXNet/t/test_loss.t
@@ -84,7 +84,7 @@ sub test_ce_loss
$loss = mx->sym->make_loss($loss);
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
local($AI::MXNet::Logging::silent) = 1;
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam');
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 0.1);
}
@@ -104,7 +104,7 @@ sub test_bce_loss
$loss = mx->sym->make_loss($loss);
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
local($AI::MXNet::Logging::silent) = 1;
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam',
initializer=>mx->init->Xavier(magnitude=>2));
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 0.01);
@@ -138,7 +138,7 @@ sub test_kl_loss
$loss = mx->sym->make_loss($loss);
local($AI::MXNet::Logging::silent) = 1;
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam');
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 0.05);
}
@@ -158,7 +158,7 @@ sub test_l2_loss
$loss = mx->sym->make_loss($loss);
local($AI::MXNet::Logging::silent) = 1;
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam');
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 0.1);
}
@@ -178,7 +178,7 @@ sub test_l1_loss
$loss = mx->sym->make_loss($loss);
local($AI::MXNet::Logging::silent) = 1;
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam');
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 0.1);
}
@@ -227,7 +227,7 @@ sub test_ctc_loss_train
$loss = mx->sym->make_loss($loss);
local($AI::MXNet::Logging::silent) = 1;
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label']);
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
1},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
1},
initializer=>mx->init->Xavier(magnitude=>2),
eval_metric=>mx->metric->Loss(),
optimizer=>'adam');
ok($mod->score($data_iter, mx->metric->Loss())->{loss} < 10);
@@ -255,7 +255,7 @@ sub test_sample_weight_loss
$loss = mx->sym->make_loss($loss);
local($AI::MXNet::Logging::silent) = 1;
my $mod = mx->mod->Module($loss, data_names=>['data'],
label_names=>['label', 'w']);
-$mod->fit($data_iter, num_epoch=>50, optimizer_params=>{learning_rate =>
0.01},
+$mod->fit($data_iter, num_epoch=>200, optimizer_params=>{learning_rate =>
0.01},
eval_metric=>mx->metric->Loss(), optimizer=>'adam');
$data_iter = mx->io->NDArrayIter(
$data->slice([10,$data->len-1]),
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
[incubator-mxnet] branch master updated: Delete workspace before build. (#8195)
This is an automated email from the ASF dual-hosted git repository.
indhub pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 573a010 Delete workspace before build. (#8195)
573a010 is described below
commit 573a010879583885a0193e30dc0b8c848d80869b
Author: Gautam Kumar <gaku...@fb.com>
AuthorDate: Mon Oct 9 18:20:12 2017 -0700
Delete workspace before build. (#8195)
* Delete workspace before build.
* Move 'deleteDir' outside the retry.
---
Jenkinsfile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Jenkinsfile b/Jenkinsfile
index 50398ce..14d1ce4 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -13,6 +13,7 @@ err = null
// initialize source codes
def init_git() {
+ deleteDir()
retry(5) {
try {
timeout(time: 2, unit: 'MINUTES') {
@@ -29,6 +30,7 @@ def init_git() {
}
def init_git_win() {
+ deleteDir()
retry(5) {
try {
timeout(time: 2, unit: 'MINUTES') {
--
To stop receiving notification emails like this one, please contact
['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].
