[
https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233947#comment-14233947
]
Nathan Marz edited comment on STORM-561 at 12/4/14 6:36 AM:
------------------------------------------------------------
[~amontalenti] Why are you using the Clojure DSL for streamparse? I think
you'll find things a lot easier to just create the Thrift objects directly in
Python code. Storm is designed so that you don't even have to touch Java or any
JVM language in order to construct and submit topologies.
Storm already has the ability to create topologies completely dynamically,
where we define "dynamically" as not needing any recompilation. Storm's Thrift
interfaces allow you to:
1. Create spout and bolt object of any language implementation from any other
language. The ComponentObject struct can be specified either as a serialized
java object, a java class name + arguments, or as a ShellComponent (for running
spouts/bolts via subprocesses to allow for other languages).
2. Define topologies from any language. Storm gets this for free by the nature
of topologies being Thrift objects.
3. Submit topologies from any language. Again, Storm gets this for free by the
nature of Nimbus being a Thrift server and topologies being Thrift objects.
This means that the majority of what's been proposed in this issue is redundant
with what's already in Storm. The exception is Trident, but that should have
its own issue opened for this feature.
The direction I'd like to see a patch for this issue go is making a nice
*library* in an interpreted language (like Python) that makes a pretty wrapper
interface around the Thrift stuff, since manipulating Thrift objects directly
is a little verbose. In addition, it can handle packaging of any artifacts the
topology will need (like spout and bolt implementations) into a .jar file. The
generated Python code for manipulating the Thrift structures is packaged with
Storm at storm-core/src/py/.
For reference, here's the storm.thrift file:
https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift
was (Author: marz):
[~amontalenti] Why are you using the Clojure DSL for streamparse? I think
you'll find things a lot easier to just create the Thrift objects directly in
Python code. Storm is designed so that you don't even have to touch Java or any
JVM language in order to construct and submit topologies.
Storm already has the ability to create topologies completely dynamically,
where we define "dynamically" as not needing any recompilation. Storm's Thrift
interfaces allow you to:
1. Create spout and bolt object of any language implementation from any other
language. The ComponentObject struct can be specified either as a serialized
java object, a java class name + arguments, or as a ShellComponent (for running
spouts/bolts via subprocesses to allow for other languages).
2. Define topologies from any language. Storm gets this for free by the nature
of topologies being Thrift objects.
3. Submit topologies from any language. Again, Storm gets this for free by the
nature of Nimbus being a Thrift server and topologies being Thrift objects.
This means that the majority of what's been proposed in this issue is redundant
with what's already in Storm. The exception is Trident, but that should have
its own issue opened for this feature.
The direction I'd like to see a patch for this issue go is making a nice
*library* in an interpreted language (like Python) that makes a pretty wrapper
interface around the Thrift stuff, since manipulating Thrift objects directly
is a little verbose. In addition, it can handle packaging of any artifacts the
topology will need (like spout and bolt implementations) into a .jar file. The
generated Python code for manipulating the Thrift structures is packaged with
Storm at storm-core/src/py/.
> Add ability to create topologies dynamically
> --------------------------------------------
>
> Key: STORM-561
> URL: https://issues.apache.org/jira/browse/STORM-561
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Nathan Leung
> Assignee: Nathan Leung
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of
> requiring a recompile to change parameters (e.g. number of workers, number of
> tasks, layout, etc).
> I would propose the following data structures for building core storm
> topologies. I haven't done a design for trident yet but the intention would
> be to add trident support when core storm support is complete (or in parallel
> if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
> String argumentType; // Class used to lookup arguments in
> method/constructor
> String implementationType; // Class used to create this argument
> String value; // String used to construct this argument
> List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
> String upstreamComponent; // name of upstream component
> String grouping;
> List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
> String name;
> String klazz; // Class of this spout
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> }
> class StormBolt {
> String name;
> String klazz; // Class of this bolt
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
> String name;
> List<StormSpout> spouts;
> List<StormBolt> bolts;
> Map config;
> int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above. The
> benefits:
> * Dependency free. Code to unmarshal from json, xml, etc, can be kept in
> extensions, or as examples, and users can write a different unmarshaller if
> they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
