[
https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230957#comment-14230957
]
Nathan Leung commented on STORM-561:
------------------------------------
Thanks for the offer. The initial implementation is almost done, I just need
to rework how I construct arguments to better support containers and arrays
(which is required to create the fieldsGrouping). Also I would like to solicit
feedback on how I'm planning to handle arguments.
Currently class Argument works the following way:
{code}
class Argument {
String argumentType; // Class used to lookup arguments in
method/constructor
String implementationType; // Class used to create this argument
String value; // String used to construct this argument
List<Argument> arguments; // arguments used to build this argument
}
{code}
- argumentType - this is the type that gets created. The argument creator will
automatically search for constructors using supertypes. This should be a fully
qualified class name. Arrays are represented by the fulling qualified class
name + "[]" (e.g. "java.lang.Integer[]"), though I would consider changing this
to use the java class name of the array type (e.g. "[Ljava.lang.Integer;").
- value - If argumentType and implementationType are not specified, it is
assumed that the argumentType is java.lang.String, and the value is passed
as-is. If this is a typical object and this string is specified, the argument
builder will look for a constructor that takes a single string as an input
parameter. If this argument is an array, then "value" will be used to create
an element that will populate the array. Also, if argumentType is a container,
I would consider letting "value" be used as a string that populates the
container, but I can see how this might be confusing.
- arguments - Only used if "value" is not specified. If this is a typical
object, then this will represent the input arguments for the constructor. If
this argument is an array or a container, then this list represents the
elements that will be used to populate this argument object.
If "value" and "arguments" are both not specified, the default constructor will
be used.
Please let me know if you have any thoughts on how the arguments work, I think
it's the most confusing (and verbose) part of the system. You can see an
example WordCountTopology below.
{code}
{
"name" : "DynamicWordCountTopology",
"spouts" : [
{
"name" : "RandomSentenceSpout",
"klazz" : "storm.starter.spout.RandomSentenceSpout",
"numTasks" : 2,
"maxPending" : 128
}
],
"bolts" : [
{
"name" : "SplitSentenceBolt",
"klazz" : "storm.starter.WordCountTopology$SplitSentence",
"numTasks" : 4,
"dependencies": [
{
"grouping": "shuffleGrouping",
"arguments": [ { "value" : "RandomSentenceSpout" } ]
}
]
},
{
"name" : "WordCountBolt",
"klazz" : "storm.starter.WordCountTopology$WordCount",
"numTasks" : 2,
"dependencies": [
{
"grouping": "fieldsGrouping",
"arguments": [
{ "value" : "SplitSentenceBolt" },
{
"argumentType" : "backtype.storm.tuple.Fields",
"arguments" : [
{
"argumentType" : "java.util.ArrayList",
"arguments" : [ { "value" : "split" } ]
}
]
}
]
}
]
}
],
"numWorkers" : 1
}
{code}
> Add ability to create topologies dynamically
> --------------------------------------------
>
> Key: STORM-561
> URL: https://issues.apache.org/jira/browse/STORM-561
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Nathan Leung
> Assignee: Nathan Leung
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of
> requiring a recompile to change parameters (e.g. number of workers, number of
> tasks, layout, etc).
> I would propose the following data structures for building core storm
> topologies. I haven't done a design for trident yet but the intention would
> be to add trident support when core storm support is complete (or in parallel
> if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
> String argumentType; // Class used to lookup arguments in
> method/constructor
> String implementationType; // Class used to create this argument
> String value; // String used to construct this argument
> List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
> String upstreamComponent; // name of upstream component
> String grouping;
> List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
> String name;
> String klazz; // Class of this spout
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> }
> class StormBolt {
> String name;
> String klazz; // Class of this bolt
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
> String name;
> List<StormSpout> spouts;
> List<StormBolt> bolts;
> Map config;
> int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above. The
> benefits:
> * Dependency free. Code to unmarshal from json, xml, etc, can be kept in
> extensions, or as examples, and users can write a different unmarshaller if
> they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
