[
https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242961#comment-14242961
]
David Willcox edited comment on STORM-561 at 12/11/14 7:01 PM:
---------------------------------------------------------------
I just found this issue and figured I'd chime in. We have something similar to
this that we use for a number of topologies in Yahoo that let us define a
topology in a YAML. (I don't want to start a json/YAML religious war, but I
find YAML to be more human-friendly, and there are things you can do in YAML
that you can't do in json.)
A very simple example of what it looks like:
spouts:
DemoSpout:
builder: com.demo.builder:spoutBuilder
parallelism: 2
someParameterForTheSpout: /usr/var/inputfile # I don't know, whatever
you need here.
bolts:
FirstBolt:
builder: com.demo.builder:boltBuilder
parallelism: 10
someBoltSpecificParameter: foo
inputs: [DemoSpout:stream1]
SecondBolt:
class: com.demo.ABolt # This one implements its own constructor
parallelism: 1
inputs: [DemoSpout:stream2, FirstBolt]
Very little detail here, of course. Just a teaser. But this exists, and is
being used in production topologies.
(OK, I need to figure out how to format this better.)
was (Author: dwillcox):
I just found this issue and figured I'd chime in. We have something similar to
this that we use for a number of topologies in Yahoo that let us define a
topology in a YAML. (I don't want to start a json/YAML religious war, but I
find YAML to be more human-friendly, and there are things you can do in YAML
that you can't do in json.)
A very simple example of what it looks like:
spouts:
DemoSpout:
builder: com.demo.builder:spoutBuilder
parallelism: 2
someParameterForTheSpout: /usr/var/inputfile # I don't know, whatever
you need here.
bolts:
FirstBolt:
builder: com.demo.builder:boltBuilder
parallelism: 10
someBoltSpecificParameter: foo
inputs: [DemoSpout:stream1]
SecondBolt:
class: com.demo.ABolt # This one implements its own constructor
parallelism: 1
inputs: [DemoSpout:stream2, FirstBolt]
Very little detail here, of course. Just a teaser. But this exists, and is
being used in production topologies.
> Add ability to create topologies dynamically
> --------------------------------------------
>
> Key: STORM-561
> URL: https://issues.apache.org/jira/browse/STORM-561
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Nathan Leung
> Assignee: Nathan Leung
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of
> requiring a recompile to change parameters (e.g. number of workers, number of
> tasks, layout, etc).
> I would propose the following data structures for building core storm
> topologies. I haven't done a design for trident yet but the intention would
> be to add trident support when core storm support is complete (or in parallel
> if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
> String argumentType; // Class used to lookup arguments in
> method/constructor
> String implementationType; // Class used to create this argument
> String value; // String used to construct this argument
> List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
> String upstreamComponent; // name of upstream component
> String grouping;
> List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
> String name;
> String klazz; // Class of this spout
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> }
> class StormBolt {
> String name;
> String klazz; // Class of this bolt
> List <Argument> arguments;
> int numTasks;
> int numExecutors;
> List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
> String name;
> List<StormSpout> spouts;
> List<StormBolt> bolts;
> Map config;
> int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above. The
> benefits:
> * Dependency free. Code to unmarshal from json, xml, etc, can be kept in
> extensions, or as examples, and users can write a different unmarshaller if
> they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
