[
https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243304#comment-14243304
]
Nathan Marz commented on STORM-561:
-----------------------------------
Topologies are Thrift objects no matter what language you use. This issue is
simply about submitting topologies without having to do a compilation first.
There are two aspects to that. First, it needs to be possible with Storm. This
is and has been the case for a long time as I've outlined above. Although it's
currently possible in Storm, it's not "pretty" because the Thrift API is
verbose.
The second is making a nice way of specifying those topologies. The proposals
here are to specify those topologies via a YAML, JSON, or other "configuration"
file – essentially, an external DSL. I am strongly against external DSLs in
general. What inevitably will happen is people will want more "dynamic" control
over the topology. For example, someone may want to specify some bolts inside
of a loop. But since the topology is specified in a non-general purpose
language, this isn't going to be possible. So to make this possible people will
want general purpose language features added to the external DSL. Going down
this road is riddled with completely unnecessary complexity.
What I'm proposing instead is to ditch the idea of specifying topologies via
configuration files and do it instead via an interpreted general purpose
programming language (like Python). By using an interpreted language, you can
construct and submit topologies without having to do a compilation, which is
the entire purpose of this issue. *You can use Java spouts and bolts in Python
just as easily as you can within Java or from within a YAML or JSON file*. I
guarantee you can make a library for specifying topologies that's as nice as
doing so via configuration files, except you never lose the power of a general
purpose programming language.
> Add ability to create topologies dynamically
> --------------------------------------------
>
> Key: STORM-561
> URL: https://issues.apache.org/jira/browse/STORM-561
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Nathan Leung
> Assignee: Nathan Leung
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of
> requiring a recompile to change parameters (e.g. number of workers, number of
> tasks, layout, etc).
> I would propose the following data structures for building core storm
> topologies. I haven't done a design for trident yet but the intention would
> be to add trident support when core storm support is complete (or in parallel
> if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
> String argumentType; // Class used to lookup arguments in
> method/constructor
> String implementationType; // Class used to create this argument
> String value; // String used to construct this argument
> List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
> String upstreamComponent; // name of upstream component
> String grouping;
> List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
> String name;
> String klazz; // Class of this spout
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> }
> class StormBolt {
> String name;
> String klazz; // Class of this bolt
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
> String name;
> List<StormSpout> spouts;
> List<StormBolt> bolts;
> Map config;
> int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above. The
> benefits:
> * Dependency free. Code to unmarshal from json, xml, etc, can be kept in
> extensions, or as examples, and users can write a different unmarshaller if
> they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
