[ https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nathan Leung updated STORM-561: ------------------------------- Description: It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc). I would propose the following data structures for building core storm topologies. I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it): {code} // fields value and arguments are mutually exclusive class Argument { String argumentType; // Class used to lookup arguments in method/constructor String implementationType; // Class used to create this argument String value; // String used to construct this argument List<Argument> arguments; // arguments used to build this argument } class Dependency { String upstreamComponent; // name of upstream component String grouping; List<Argument> arguments; // arguments for the grouping } class StormSpout { String name; String klazz; // Class of this spout List <Argument> arguments; int numTasks; int numExecutors; } class StormBolt { String name; String klazz; // Class of this bolt List <Argument> arguments; int numTasks; int numExecutors; List<Dependency> dependencies; } class StormTopologyRepresentation { String name; List<StormSpout> spouts; List<StormBolt> bolts; Map config; int numWorkers; } {code} Topology creation will be built on top of the data structures above. The benefits: * Dependency free. Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation. * support arbitrary spout and bolts types * support of all groupings, streams, via reflections * ability to specify configuration map via config file * reification of spout / bolt / dependency arguments ** recursive argument reification for complex objects was: It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc). I would propose the following data structures for building core storm topologies. I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it): {code} // fields value and arguments are mutually exclusive class Argument { Class argumentType; // type used to lookup arguments in method/constructor Class implementationType; // type used to create this argument String value; // String used to construct this argument List<Argument> arguments; // arguments used to build this argument } class Dependency { String upstreamComponent; // name of upstream component String grouping; List<Argument> arguments; // arguments for the grouping } class StormSpout { String name; Class class; // class of this spout int numTasks; int numExecutors; } class StormBolt { String name; Class class; int numTasks; int numExecutors; List<Dependency> dependencies; } class StormTopologyRepresentation { String name; List<StormSpout> spouts; List<StormBolt> bolts; Map config; int numWorkers; } {code} Topology creation will be built on top of the data structures above. The benefits: * Dependency free. Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation. * support arbitrary spout and bolts types * support of all groupings, streams, via reflections * ability to specify configuration map via config file * reification of spout / bolt / dependency arguments ** recursive argument reification for complex objects > Add ability to create topologies dynamically > -------------------------------------------- > > Key: STORM-561 > URL: https://issues.apache.org/jira/browse/STORM-561 > Project: Apache Storm > Issue Type: Improvement > Reporter: Nathan Leung > Assignee: Nathan Leung > Original Estimate: 336h > Remaining Estimate: 336h > > It would be nice if a storm topology could be built dynamically, instead of > requiring a recompile to change parameters (e.g. number of workers, number of > tasks, layout, etc). > I would propose the following data structures for building core storm > topologies. I haven't done a design for trident yet but the intention would > be to add trident support when core storm support is complete (or in parallel > if there are other people working on it): > {code} > // fields value and arguments are mutually exclusive > class Argument { > String argumentType; // Class used to lookup arguments in > method/constructor > String implementationType; // Class used to create this argument > String value; // String used to construct this argument > List<Argument> arguments; // arguments used to build this argument > } > class Dependency { > String upstreamComponent; // name of upstream component > String grouping; > List<Argument> arguments; // arguments for the grouping > } > class StormSpout { > String name; > String klazz; // Class of this spout > List <Argument> arguments; > int numTasks; > int numExecutors; > } > class StormBolt { > String name; > String klazz; // Class of this bolt > List <Argument> arguments; > int numTasks; > int numExecutors; > List<Dependency> dependencies; > } > class StormTopologyRepresentation { > String name; > List<StormSpout> spouts; > List<StormBolt> bolts; > Map config; > int numWorkers; > } > {code} > Topology creation will be built on top of the data structures above. The > benefits: > * Dependency free. Code to unmarshal from json, xml, etc, can be kept in > extensions, or as examples, and users can write a different unmarshaller if > they want to use a different text representation. > * support arbitrary spout and bolts types > * support of all groupings, streams, via reflections > * ability to specify configuration map via config file > * reification of spout / bolt / dependency arguments > ** recursive argument reification for complex objects -- This message was sent by Atlassian JIRA (v6.3.4#6332)