[
https://issues.apache.org/jira/browse/SINGA-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623318#comment-14623318
]
ASF subversion and git services commented on SINGA-29:
------------------------------------------------------
Commit 9a6e09fa2e56ea4c2563264c378ee9f3eb314acf in incubator-singa's branch
refs/heads/master from wang wei
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=9a6e09f ]
SINGA-29 Update NeuralNet class to enable customizing layer partition type
1. Clean the code for NeuralNet and Graph classes.
Graph class only provides functions about Node and Edge management, e.g.,
add, remove and toplogy sort.
NeuralNet provides one function (CreateGraph) to convert net configuration
into a Graph. Net partitioning
is done in CreateGraph function. The CreateNetFromGraph function create and
connect layers from the graph.
2. Users can customize the partition for whole net and for a specific layer
through field partition_dim of LayerProto and NetProto.
the configuration of LayerProto overwrites that of NetProto.
Tested on single process with non-distributed training, shared-memory hogwild,
one worker group with 2 workers.
Tested with two processes for downpour and distributed hogwild. Downpour has
similar performance as shared-memory hoglwild; while Distributed hogwild does
not perform as good non-distributed training.
> Update NeuralNet class to enable layer partition type customization
> -------------------------------------------------------------------
>
> Key: SINGA-29
> URL: https://issues.apache.org/jira/browse/SINGA-29
> Project: Singa
> Issue Type: Bug
> Reporter: wangwei
> Assignee: wangwei
>
> This ticket is to update the NeuralNet class to enable users customize the
> partitioning of each layer. It also cleans the code for NeuralNet class and
> Graph class.
> The are two places where the user can configure the partitioning of the
> neural net.
> * partition_dim for the whole neural net (in NetProto)
> * partition_dim for each layer (in LayerProto)
> The partition_dim of the net will be copied to each layer if the layer's
> partition_dim is not set; Otherwise, the layer's own partition_dim will be
> used.
> Currently we support three values of partition_dim:
> * partition_dim = -1, no partition
> * partition_dim = 0, partition along the batch dimension, e.g., partition one
> mini-batch of 100 images into two partitions, each with 50 images.
> * partition_dim = 1, partition along feature dimension, e.g., if we partition
> one mini-batch of 100 images, each represented using 128-d feature vector,
> into two partitions. Each partition would have 100 images, each represented
> using 64-d feature vector.
> Simple partitioning schemes will be tested firstly, e.g., partition the whole
> net with partition_dim = 0. Complex partitioning scheme, e.g., hybrid
> partitioning (some layers are partitioned along dimension 0 and some layers
> are partitioned along dimension 1 or -1) will be tested later. Advanced
> partition like assigning some layers to one worker and other layers to
> another worker will be supported by customizing partition_id of LayerProto in
> the future.
> NeuralNet is constructed as follows:
> Neural net configuration is converted to a graph with one node per (sub)
> layer. Some connection nodes will be inserted automatically if the neural net
> needs partitioning (e.g., group size >1). After topology sort, one Layer will
> be created per node and layers will be connected accordingly. The Graph class
> provides functions for adding/removing nodes and edges, and sorting nodes in
> topology order. Each node stores the configuration of one layer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
