[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250897378 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py --n_process $(nproc) +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - tgt_path : Path for preprocessed images (default='./data/datasets/') + - n_process : num of process (default=1) + +You can run the preprocessing with just one processor, but this will take a long time (>48 hours). To use all of the available processors, use the following command: + +``` +cd ./utils && python preprocess_data.py --n_process $(nproc) +``` + +## Data Structure + +``` +The training data folder should look like : + +|--datasets +|--s1 + |--bbir7s + |--mouth_000.png + |--mouth_001.png + ... + |--bgaa8p + |--mouth_000.png + |--mouth_001.png + ... +|--s2 +... + |--align + |--bw1d8a.align + |--bggzzs.align + ... + +``` + + +## Training + Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250897289
##
File path: example/gluon/lipnet/lipnet_model.ipynb
##
@@ -0,0 +1,249 @@
+{
Review comment:
@thomelane @seujung I Removed the ipynb. I made a new file(infer.py) for
inference and refactored trainer.py
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250897345
##
File path: example/gluon/lipnet/lipnet_model.ipynb
##
@@ -0,0 +1,249 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+"import sys\n",
+"import mxnet as mx\n",
+"from mxnet import gluon, autograd ,nd\n",
+"import argparse\n",
+"from trainer import setting_ctx, Train, char_beam_search\n",
+"from data_loader import LipsDataset\n",
+"from models.network import LipNet\n",
+"from mxnet.gluon.data.vision import transforms\n",
+"from tqdm import tqdm, trange\n",
Review comment:
Deleted the ipynb
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250896405 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py --n_process $(nproc) +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - tgt_path : Path for preprocessed images (default='./data/datasets/') + - n_process : num of process (default=1) + +You can run the preprocessing with just one processor, but this will take a long time (>48 hours). To use all of the available processors, use the following command: + +``` +cd ./utils && python preprocess_data.py --n_process $(nproc) +``` + +## Data Structure + +``` +The training data folder should look like : + +|--datasets +|--s1 + |--bbir7s + |--mouth_000.png + |--mouth_001.png + ... + |--bgaa8p + |--mouth_000.png + |--mouth_001.png + ... +|--s2 +... + |--align + |--bw1d8a.align + |--bggzzs.align + ... + +``` + + +## Training + +- arguments + - batch_size : Define batch size (default=64) + - epochs : Define total epochs (default=100) + - image_path : Path for lip image files (default='./data/datasets/') + - align_path : Path for align files (default='./data/align/') + - dr_rate : Dropout rate(default=0.5) + - num_gpus : Num of gpus (if num_gpus is 0, then use cpu) (default=1) + - num_workers : Num of workers when generating data (default=0) + +``` +python main.py +``` + +## Results +``` Review comment: Remove iypnb and Add the infer.py file This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250896251 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py --n_process $(nproc) +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments Review comment: Update more infomation in Readme This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250835129 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : LipNet module using gluon +""" + + +from mxnet import nd +from mxnet.gluon import nn, rnn +import mxnet.ndarray as F +# pylint: disable=too-many-instance-attributes +class LipNet(nn.Block): +""" +Description : LipNet network using gluon +""" +def __init__(self, dr_rate, **kwargs): +super(LipNet, self).__init__(**kwargs) +with self.name_scope(): +self.conv1 = nn.Conv3D(32, kernel_size=(3, 5, 5), strides=(1, 2, 2), padding=(1, 2, 2)) +#self.bn1 = nn.BatchNorm() +self.bn1 = nn.InstanceNorm(in_channels=32) +self.dr1 = nn.Dropout(dr_rate) +self.pool1 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.conv2 = nn.Conv3D(64, kernel_size=(3, 5, 5), strides=(1, 1, 1), padding=(1, 2, 2)) +#self.bn2 = nn.BatchNorm() +self.bn2 = nn.InstanceNorm(in_channels=64) +self.dr2 = nn.Dropout(dr_rate) +self.pool2 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.conv3 = nn.Conv3D(96, kernel_size=(3, 3, 3), strides=(1, 1, 1), padding=(1, 2, 2)) +#self.bn3 = nn.BatchNorm() +self.bn3 = nn.InstanceNorm(in_channels=96) +self.dr3 = nn.Dropout(dr_rate) +self.pool3 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.gru1 = rnn.GRU(256, bidirectional=True) +self.gru2 = rnn.GRU(256, bidirectional=True) +self.dense = nn.Dense(27+1, flatten=False) +# pylint: disable=arguments-differ +def forward(self, x): +out = self.conv1(x) +out = self.bn1(out) +out = F.relu(out) +out = self.dr1(out) +out = self.pool1(out) +out = self.conv2(out) +out = self.bn2(out) +out = F.relu(out) +out = self.dr2(out) +out = self.pool2(out) +out = self.conv3(out) +out = self.bn3(out) +out = F.relu(out) +out = self.dr3(out) +out = self.pool3(out) +out = nd.transpose(out, (2, 0, 1, 3, 4)) +# pylint: disable=no-member +out = out.reshape((out.shape[0], out.shape[1], -1)) Review comment: I changed ``` out = out.reshape((0, 0, -1)) ``` it it easy to apply in the case of multi-gpu. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250465119 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : LipNet module using gluon +""" + + +from mxnet import nd +from mxnet.gluon import nn, rnn +import mxnet.ndarray as F +# pylint: disable=too-many-instance-attributes +class LipNet(nn.Block): +""" +Description : LipNet network using gluon +""" +def __init__(self, dr_rate, **kwargs): +super(LipNet, self).__init__(**kwargs) +with self.name_scope(): +self.conv1 = nn.Conv3D(32, kernel_size=(3, 5, 5), strides=(1, 2, 2), padding=(1, 2, 2)) +#self.bn1 = nn.BatchNorm() +self.bn1 = nn.InstanceNorm(in_channels=32) +self.dr1 = nn.Dropout(dr_rate) +self.pool1 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.conv2 = nn.Conv3D(64, kernel_size=(3, 5, 5), strides=(1, 1, 1), padding=(1, 2, 2)) +#self.bn2 = nn.BatchNorm() +self.bn2 = nn.InstanceNorm(in_channels=64) +self.dr2 = nn.Dropout(dr_rate) +self.pool2 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.conv3 = nn.Conv3D(96, kernel_size=(3, 3, 3), strides=(1, 1, 1), padding=(1, 2, 2)) +#self.bn3 = nn.BatchNorm() +self.bn3 = nn.InstanceNorm(in_channels=96) +self.dr3 = nn.Dropout(dr_rate) +self.pool3 = nn.MaxPool3D((1, 2, 2), (1, 2, 2)) +self.gru1 = rnn.GRU(256, bidirectional=True) +self.gru2 = rnn.GRU(256, bidirectional=True) +self.dense = nn.Dense(27+1, flatten=False) +# pylint: disable=arguments-differ +def forward(self, x): +out = self.conv1(x) +out = self.bn1(out) +out = F.relu(out) +out = self.dr1(out) +out = self.pool1(out) +out = self.conv2(out) +out = self.bn2(out) +out = F.relu(out) +out = self.dr2(out) +out = self.pool2(out) +out = self.conv3(out) +out = self.bn3(out) +out = F.relu(out) +out = self.dr3(out) +out = self.pool3(out) +out = nd.transpose(out, (2, 0, 1, 3, 4)) +# pylint: disable=no-member +out = out.reshape((out.shape[0], out.shape[1], -1)) Review comment: I didn't konw what 0 means. So I changed the code. ``` out = out.reshape((self.seq_len, self.batch_size, -1)) ``` It also works when the code is changed like this. ``` out = out.reshape((0, 0, -1)) ``` If you want to change ```0```, I will do. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250465435 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : LipNet module using gluon +""" + + +from mxnet import nd +from mxnet.gluon import nn, rnn +import mxnet.ndarray as F Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250463199 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : LipNet module using gluon +""" + + +from mxnet import nd +from mxnet.gluon import nn, rnn +import mxnet.ndarray as F +# pylint: disable=too-many-instance-attributes +class LipNet(nn.Block): Review comment: Hybridized This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250447022 ## File path: example/gluon/lipnet/trainer.py ## @@ -0,0 +1,152 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : Training module for LipNet +""" + + +import sys +import mxnet as mx +from mxnet import gluon, autograd, nd +from mxnet.gluon.data.vision import transforms +from tqdm import tqdm, trange +from data_loader import LipsDataset +from models.network import LipNet +from BeamSearch import ctcBeamSearch +from utils.common import char_conv, int2char +# set gpu count + + +def setting_ctx(num_gpus): +""" +Description : set gpu module +""" +if num_gpus > 0: +ctx = [mx.gpu(i) for i in range(num_gpus)] +else: +ctx = [mx.cpu()] +return ctx + + +ALPHABET = '' +for i in range(27): +ALPHABET += int2char(i) + +def char_beam_search(out): +""" +Description : apply beam search for prediction result +""" +out_conv = list() +for idx in range(out.shape[0]): +probs = out[idx] +prob = probs.softmax().asnumpy() +line_string_proposals = ctcBeamSearch(prob, ALPHABET, None, k=4, beamWidth=25) +out_conv.append(line_string_proposals[0]) +return out_conv + +# pylint: disable=too-many-instance-attributes, too-many-locals +class Train: +""" +Description : Train class for training network +""" +def __init__(self, config): +##setting hyper-parameters +self.batch_size = config.batch_size +self.epochs = config.epochs +self.image_path = config.image_path +self.align_path = config.align_path +self.dr_rate = config.dr_rate +self.num_gpus = config.num_gpus +self.ctx = setting_ctx(self.num_gpus) +self.num_workers = config.num_workers +self.build_model() + +def build_model(self): +""" +Description : build network +""" +#set network +self.net = LipNet(self.dr_rate) +self.net.initialize(ctx=self.ctx) +#set optimizer +self.loss_fn = gluon.loss.CTCLoss() +self.trainer = gluon.Trainer(self.net.collect_params(), \ + optimizer='SGD') +def save_model(self, epoch, iter_no, current_loss): +""" +Description : save parameter of network weight +""" +file_name = "checkpoint/best_model_epoches_"+str(epoch)+"iter_"+str(iter_no)+ \ +"loss_"+str(round(current_loss, 2)) +self.net.save_parameters(file_name) + +def train(self): +""" +Description : training for LipNet +""" +input_transform = transforms.Compose([transforms.ToTensor(), \ + transforms.Normalize((0.7136, 0.4906, 0.3283), \ + (0.1138, 0.1078, 0.0917))]) +training_dataset = LipsDataset(self.image_path, self.align_path, transform=input_transform) +train_dataloader = mx.gluon.data.DataLoader(training_dataset, +batch_size=self.batch_size, +shuffle=True, + num_workers=self.num_workers) +best_loss = sys.maxsize +for epoch in trange(self.epochs): +iter_no = 0 +for input_data, input_label in tqdm(train_dataloader): +input_data = nd.transpose(input_data, (0, 2, 1, 3, 4)) Review comment: Moved into data_loader This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250446948 ## File path: example/gluon/lipnet/data_loader.py ## @@ -0,0 +1,73 @@ +""" +Description : Set DataSet module for lip images +""" +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import os +import glob +from mxnet import nd +import mxnet.gluon.data.dataset as dataset +from mxnet.gluon.data.vision.datasets import image +from utils.align import Align + +class LipsDataset(dataset.Dataset): +""" +Description : DataSet class for lip images +""" +def __init__(self, root, align_root, flag=1, transform=None): +self._root = os.path.expanduser(root) +self._align_root = align_root +self._flag = flag +self._transform = transform +self._exts = ['.jpg', '.jpeg', '.png'] +self._list_images(self._root) + +def _list_images(self, root): +""" +Description : generate list for lip images +""" +self.labels = [] +self.items = [] +folder_path = glob.glob(os.path.join(root, "*", "*")) +for folder in folder_path: +filename = glob.glob(os.path.join(folder, "*")) +if len(filename) != 75: +continue +filename.sort() +label = os.path.split(folder)[-1] +self.items.append((filename, label)) +def align_generation(self, file_nm, padding=75): +""" +Description : Align to lip position +""" +align = Align(self._align_root + '/' + file_nm + '.align') +return nd.array(align.sentence(padding)) +def __getitem__(self, idx): +img = list() +for image_name in self.items[idx][0]: +tmp_img = image.imread(image_name, self._flag) +if self._transform is not None: +tmp_img = self._transform(tmp_img) +img.append(tmp_img) +img = nd.stack(*img) +#print(self.items[idx][0][0]) Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250219354 ## File path: example/gluon/lipnet/trainer.py ## @@ -0,0 +1,152 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : Training module for LipNet +""" + + +import sys +import mxnet as mx +from mxnet import gluon, autograd, nd +from mxnet.gluon.data.vision import transforms +from tqdm import tqdm, trange +from data_loader import LipsDataset +from models.network import LipNet +from BeamSearch import ctcBeamSearch +from utils.common import char_conv, int2char +# set gpu count + + +def setting_ctx(num_gpus): +""" +Description : set gpu module +""" +if num_gpus > 0: +ctx = [mx.gpu(i) for i in range(num_gpus)] +else: +ctx = [mx.cpu()] +return ctx + + +ALPHABET = '' +for i in range(27): +ALPHABET += int2char(i) + +def char_beam_search(out): +""" +Description : apply beam search for prediction result +""" +out_conv = list() +for idx in range(out.shape[0]): +probs = out[idx] +prob = probs.softmax().asnumpy() +line_string_proposals = ctcBeamSearch(prob, ALPHABET, None, k=4, beamWidth=25) +out_conv.append(line_string_proposals[0]) +return out_conv + +# pylint: disable=too-many-instance-attributes, too-many-locals +class Train: +""" +Description : Train class for training network +""" +def __init__(self, config): +##setting hyper-parameters +self.batch_size = config.batch_size +self.epochs = config.epochs +self.image_path = config.image_path +self.align_path = config.align_path +self.dr_rate = config.dr_rate +self.num_gpus = config.num_gpus +self.ctx = setting_ctx(self.num_gpus) +self.num_workers = config.num_workers +self.build_model() + +def build_model(self): +""" +Description : build network +""" +#set network +self.net = LipNet(self.dr_rate) +self.net.initialize(ctx=self.ctx) +#set optimizer +self.loss_fn = gluon.loss.CTCLoss() +self.trainer = gluon.Trainer(self.net.collect_params(), \ + optimizer='SGD') +def save_model(self, epoch, iter_no, current_loss): +""" +Description : save parameter of network weight +""" +file_name = "checkpoint/best_model_epoches_"+str(epoch)+"iter_"+str(iter_no)+ \ +"loss_"+str(round(current_loss, 2)) +self.net.save_parameters(file_name) + +def train(self): +""" +Description : training for LipNet +""" +input_transform = transforms.Compose([transforms.ToTensor(), \ + transforms.Normalize((0.7136, 0.4906, 0.3283), \ + (0.1138, 0.1078, 0.0917))]) +training_dataset = LipsDataset(self.image_path, self.align_path, transform=input_transform) +train_dataloader = mx.gluon.data.DataLoader(training_dataset, +batch_size=self.batch_size, +shuffle=True, + num_workers=self.num_workers) +best_loss = sys.maxsize +for epoch in trange(self.epochs): +iter_no = 0 +for input_data, input_label in tqdm(train_dataloader): +input_data = nd.transpose(input_data, (0, 2, 1, 3, 4)) +data = gluon.utils.split_and_load(input_data, self.ctx, even_split=False) +label = gluon.utils.split_and_load(input_label, self.ctx, even_split=False) + +# pylint: disable=no-member +losses = [] +sum_losses = 0 +len_losses = 0 +with autograd.record(): +with autograd.train_mode(): +for X, Y in zip(data, label): +current_batch_loss = self.loss_fn(self.net(X), Y) +los
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250219420 ## File path: example/gluon/lipnet/trainer.py ## @@ -0,0 +1,152 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Description : Training module for LipNet +""" + + +import sys +import mxnet as mx +from mxnet import gluon, autograd, nd +from mxnet.gluon.data.vision import transforms +from tqdm import tqdm, trange +from data_loader import LipsDataset +from models.network import LipNet +from BeamSearch import ctcBeamSearch +from utils.common import char_conv, int2char +# set gpu count + + +def setting_ctx(num_gpus): +""" +Description : set gpu module +""" +if num_gpus > 0: +ctx = [mx.gpu(i) for i in range(num_gpus)] +else: +ctx = [mx.cpu()] +return ctx + + +ALPHABET = '' +for i in range(27): +ALPHABET += int2char(i) + +def char_beam_search(out): +""" +Description : apply beam search for prediction result +""" +out_conv = list() +for idx in range(out.shape[0]): +probs = out[idx] +prob = probs.softmax().asnumpy() +line_string_proposals = ctcBeamSearch(prob, ALPHABET, None, k=4, beamWidth=25) +out_conv.append(line_string_proposals[0]) +return out_conv + +# pylint: disable=too-many-instance-attributes, too-many-locals +class Train: +""" +Description : Train class for training network +""" +def __init__(self, config): +##setting hyper-parameters +self.batch_size = config.batch_size +self.epochs = config.epochs +self.image_path = config.image_path +self.align_path = config.align_path +self.dr_rate = config.dr_rate +self.num_gpus = config.num_gpus +self.ctx = setting_ctx(self.num_gpus) +self.num_workers = config.num_workers +self.build_model() + +def build_model(self): +""" +Description : build network +""" +#set network +self.net = LipNet(self.dr_rate) +self.net.initialize(ctx=self.ctx) +#set optimizer +self.loss_fn = gluon.loss.CTCLoss() +self.trainer = gluon.Trainer(self.net.collect_params(), \ + optimizer='SGD') +def save_model(self, epoch, iter_no, current_loss): Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r250213129 ## File path: example/gluon/lipnet/BeamSearch.py ## @@ -0,0 +1,168 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review comment: Added. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r246665681
##
File path: example/gluon/lipnet/BeamSearch.py
##
@@ -0,0 +1,168 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Module : this module to decode using beam search
+https://github.com/ThomasDelteil/HandwrittenTextRecognition_MXNet/blob/master/utils/CTCDecoder/BeamSearch.py
+"""
+
+from __future__ import division
+from __future__ import print_function
+import numpy as np
+
+class BeamEntry:
+"""
+information about one single beam at specific time-step
+"""
+def __init__(self):
+self.prTotal = 0 # blank and non-blank
+self.prNonBlank = 0 # non-blank
+self.prBlank = 0 # blank
+self.prText = 1 # LM score
+self.lmApplied = False # flag if LM was already applied to this beam
+self.labeling = () # beam-labeling
+
+class BeamState:
+"""
+information about the beams at specific time-step
+"""
+def __init__(self):
+self.entries = {}
+
+def norm(self):
+"""
+length-normalise LM score
+"""
+for (k, _) in self.entries.items():
+labelingLen = len(self.entries[k].labeling)
+self.entries[k].prText = self.entries[k].prText ** (1.0 /
(labelingLen if labelingLen else 1.0))
+
+def sort(self):
+"""
+return beam-labelings, sorted by probability
+"""
+beams = [v for (_, v) in self.entries.items()]
+sortedBeams = sorted(beams, reverse=True, key=lambda x:
x.prTotal*x.prText)
+return [x.labeling for x in sortedBeams]
+
+def applyLM(parentBeam, childBeam, classes, lm):
+"""
+calculate LM score of child beam by taking score from parent beam and
bigram probability of last two chars
+"""
+if lm and not childBeam.lmApplied:
+c1 = classes[parentBeam.labeling[-1] if parentBeam.labeling else
classes.index(' ')] # first char
+c2 = classes[childBeam.labeling[-1]] # second char
+lmFactor = 0.01 # influence of language model
+bigramProb = lm.getCharBigram(c1, c2) ** lmFactor # probability of
seeing first and second char next to each other
+childBeam.prText = parentBeam.prText * bigramProb # probability of
char sequence
+childBeam.lmApplied = True # only apply LM once per beam entry
+
+def addBeam(beamState, labeling):
+"""
+add beam if it does not yet exist
+"""
+if labeling not in beamState.entries:
+beamState.entries[labeling] = BeamEntry()
+
+def ctcBeamSearch(mat, classes, lm, k, beamWidth):
Review comment:
Added
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r246645057 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py --n_process $(nproc) +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - tgt_path : Path for preprocessed images (default='./data/datasets/') + - n_process : num of process (default=1) + +You can run the preprocessing with just one processor, but this will take a long time (>48 hours). To use all of the available processors, use the following command: + Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r246644994 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r246645004 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r246645027 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,115 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r245339705 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,113 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - tgt_path : Path for preprocessed images (default='./data/datasets/') + - n_process : num of process (default=1) + +``` +cd ./utils && python preprocess_data.py Review comment: Update. In addition to the preprocess_data, $(nproc) is added in the download_data. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r245339177 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,113 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon implementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 +- The Required Disk Space: 35Gb +``` +pip install -r requirements.txt +``` + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py +``` + +### Preprocess the Data: Extracting the mouth images from a video and save it. +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - tgt_path : Path for preprocessed images (default='./data/datasets/') + - n_process : num of process (default=1) + Review comment: Update This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244896251 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,110 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon inplementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 + Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244569435 ## File path: example/gluon/lipnet/README.md ## @@ -0,0 +1,110 @@ +# LipNet: End-to-End Sentence-level Lipreading + +--- + +Gluon inplementation of [LipNet: End-to-End Sentence-level Lipreading](https://arxiv.org/abs/1611.01599) + + + +## Requirements +- Python 3.6.4 +- MXnet 1.3.0 + + +## Test Environment +- 4 CPU cores +- 1 GPU (Tesla K80 12GB) + + +## The Data +- The GRID audiovisual sentence corpus (http://spandh.dcs.shef.ac.uk/gridcorpus/) +- Video: (normal)(480 M each) +- Align: word alignments(190 K each) + +## Prepare the Data +### Download the data +- arguments + - src_path : Path for videos (default='./data/mp4s/') + - align_path : Path for aligns (default='./data/align/') + - n_process : num of process (default=1) + +``` +cd ./utils && python download_data.py Review comment: I changed it to resume the download process. (1) It check if the folder such as 's1' exists (2) If there is the folder, it skips. Otherwise, it download the file and unzip the file. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244565369 ## File path: example/gluon/lipnet/BeamSearch.py ## @@ -0,0 +1,149 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from __future__ import division +from __future__ import print_function +import numpy as np + +class BeamEntry: +"information about one single beam at specific time-step" Review comment: Switched and Added url reference. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564523
##
File path: example/gluon/lipnet/utils/multi.py
##
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Module: preprocess with multi-process
+"""
+
+
+def multi_p_run(tot_num, _func, worker, params, n_process):
+"""
+Run _func with multi-process using params.
+"""
+from multiprocessing import Process, Queue
+out_q = Queue()
+procs = []
+
+split_num = split_seq(list(range(0, tot_num)), n_process)
+
+print(tot_num, ">>", split_num)
+
+split_len = len(split_num)
+if n_process > split_len:
+n_process = split_len
+
+for i in range(n_process):
+_p = Process(target=_func,
+ args=(worker, split_num[i][0], split_num[i][1],
+ params, out_q))
+_p.daemon = True
+procs.append(_p)
+_p.start()
+
+try:
+result = []
+for i in range(n_process):
+result.append(out_q.get())
+for i in procs:
+i.join()
+except KeyboardInterrupt:
+print('Killing all the childer in the pool.')
+for i in procs:
+i.terminate()
+i.join()
+return -1
+
+while not out_q.empty():
+print(out_q.get(block=False))
+
+return result
+
+
+def split_seq(sam_num, n_tile):
+"""
+Spli the number(sam_num) into numbers by n_tile
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564490 ## File path: example/gluon/lipnet/utils/common.py ## @@ -0,0 +1,80 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Module: common Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564362 ## File path: example/gluon/lipnet/trainer.py ## @@ -0,0 +1,137 @@ +""" Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564226 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,76 @@ +""" Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564231 ## File path: example/gluon/lipnet/models/network.py ## @@ -0,0 +1,76 @@ +""" +Descrition : LipNet module using gluon Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564038 ## File path: example/gluon/lipnet/main.py ## @@ -0,0 +1,41 @@ +""" +Descrition : main module to run code Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564007 ## File path: example/gluon/lipnet/main.py ## @@ -0,0 +1,41 @@ +""" Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564007 ## File path: example/gluon/lipnet/main.py ## @@ -0,0 +1,41 @@ +""" Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244564025 ## File path: example/gluon/lipnet/main.py ## @@ -0,0 +1,41 @@ +""" +Descrition : main module to run code +""" +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import argparse +from trainer import Train + +def main(): +""" +Description : run code using argument info Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244563501
##
File path: example/gluon/lipnet/lipnet_model.ipynb
##
@@ -0,0 +1,249 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+"import sys\n",
+"import mxnet as mx\n",
+"from mxnet import gluon, autograd ,nd\n",
+"import argparse\n",
+"from trainer import setting_ctx, Train, char_beam_search\n",
+"from data_loader import LipsDataset\n",
+"from models.network import LipNet\n",
+"from mxnet.gluon.data.vision import transforms\n",
+"from tqdm import tqdm, trange\n",
+"from utils.common import *"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+"## Set the argument"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+"args = dict()\n",
+"args['batch_size'] = 64\n",
+"args['epochs'] = 100\n",
+"args['image_path'] =
'/home/ubuntu/works/2018/lips_model/data/datasets/'\n",
Review comment:
Changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244285330 ## File path: example/gluon/lipnet/LICENSE ## @@ -0,0 +1,21 @@ +MIT License Review comment: Removed inappropriate license file This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example
URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244563501
##
File path: example/gluon/lipnet/lipnet_model.ipynb
##
@@ -0,0 +1,249 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+"import sys\n",
+"import mxnet as mx\n",
+"from mxnet import gluon, autograd ,nd\n",
+"import argparse\n",
+"from trainer import setting_ctx, Train, char_beam_search\n",
+"from data_loader import LipsDataset\n",
+"from models.network import LipNet\n",
+"from mxnet.gluon.data.vision import transforms\n",
+"from tqdm import tqdm, trange\n",
+"from utils.common import *"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+"## Set the argument"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+"args = dict()\n",
+"args['batch_size'] = 64\n",
+"args['epochs'] = 100\n",
+"args['image_path'] =
'/home/ubuntu/works/2018/lips_model/data/datasets/'\n",
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244285330 ## File path: example/gluon/lipnet/LICENSE ## @@ -0,0 +1,21 @@ +MIT License Review comment: Remove inappropriate license file This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] soeque1 commented on a change in pull request #13647: Update lip reading example
soeque1 commented on a change in pull request #13647: Update lip reading example URL: https://github.com/apache/incubator-mxnet/pull/13647#discussion_r244285130 ## File path: example/gluon/lipnet/.gitignore ## @@ -0,0 +1,106 @@ +# Byte-compiled / optimized / DLL files Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
