[
https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232597#comment-14232597
]
Nathan Leung commented on STORM-561:
------------------------------------
Do you mean a serialized representation of the object? Currently I've placed
the topology building code into the storm-core project in my branch (it seemed
the most appropriate at the time). With that in mind, I tried to avoid a
couple of things:
* Adding a serializer dependency to storm-core (e.g. gson)
* Putting overly complicated objects into the serialized representation. My
experience with json indicates that this often leads to a requirement for
custom serializers or deserializers and I wanted to avoid this. I guess the
arguments can be kind of clunky though.
That said, I'm not sure I'm 100% happy with how things are structured right
now. I have the topology building classes in storm-core, as well as an
unmarshalling interface that is used to pass an unmarshaller to the topology
building code. The unmarshaller implementation I created is placed into
examples, and then the user would supply their own unmarshaller and would also
be able to specify the topology in whatever data format they want (json, xml,
etc), as long as it binds to the appropriate data structures.
Here's the unmarshaller I've written that uses gson. Since it's in examples,
it does not add a dependency to storm-core. I'm also on the fence as to
whether this approach is too flexible.
{code}
package storm.starter;
import com.google.gson.Gson;
import backtype.storm.topology.dynamic.StormTopologyRepresentation;
import backtype.storm.topology.dynamic.StormTopologyUnmarshaller;
public class GsonTopologyUnmarshaller implements StormTopologyUnmarshaller {
Gson gson = new Gson();
@Override
public StormTopologyRepresentation unmarshal(String stringRepresentation) {
return gson.fromJson(stringRepresentation,
StormTopologyRepresentation.class);
}
}
{code}
> Add ability to create topologies dynamically
> --------------------------------------------
>
> Key: STORM-561
> URL: https://issues.apache.org/jira/browse/STORM-561
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Nathan Leung
> Assignee: Nathan Leung
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of
> requiring a recompile to change parameters (e.g. number of workers, number of
> tasks, layout, etc).
> I would propose the following data structures for building core storm
> topologies. I haven't done a design for trident yet but the intention would
> be to add trident support when core storm support is complete (or in parallel
> if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
> String argumentType; // Class used to lookup arguments in
> method/constructor
> String implementationType; // Class used to create this argument
> String value; // String used to construct this argument
> List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
> String upstreamComponent; // name of upstream component
> String grouping;
> List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
> String name;
> String klazz; // Class of this spout
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> }
> class StormBolt {
> String name;
> String klazz; // Class of this bolt
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
> String name;
> List<StormSpout> spouts;
> List<StormBolt> bolts;
> Map config;
> int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above. The
> benefits:
> * Dependency free. Code to unmarshal from json, xml, etc, can be kept in
> extensions, or as examples, and users can write a different unmarshaller if
> they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
