You're always going to have some amount of Flume code in process to validate Flume configs. I can understand breaking them out into a module if that's the goal, but I'd still use validators (i.e. classes called FooValidator with a single validate() method that takes a Context for each component that is configurable). That was the only point I wanted to make.
On Thu, Mar 1, 2012 at 1:25 PM, Hari Shreedharan <[email protected]>wrote: > Hi Eric, > > I had considered the validator pattern. What we are trying to do is to be > able to validate the configuration for specific components specified in the > conf, without actually depending on any of the Flume API themselves. > > The use case I am trying to address: Say you have a 100 node cluster on > which we want to install and run flume and we use some automated way of > starting up flume on these clusters, usually from one box. The idea is that > the conf file be verified before actually deploying it on the cluster > itself. Requiring Flume to be installed on these boxes(which could as well > be on the corp network) should not be a requirement. Using the validator > pattern directly would require us to actually have the flume components on > the box validating the conf. > > We could use a stub that stores all the required parameters and run the > validator on that, but I think that is pretty much the same as what is > already done here. > > Thanks! > Hari > > -- > Hari Shreedharan > > > On Thursday, March 1, 2012 at 11:20 AM, Eric Sammer wrote: > > > Why not just follow the Validator pattern? See Spring's validators for > > reference. > > > > > > > > On Feb 29, 2012, at 4:31 PM, Hari Shreedharan > > <[email protected](mailto: > [email protected])> wrote: > > > > > > > > ----------------------------------------------------------- > > > This is an automatically generated e-mail. To reply, visit: > > > https://reviews.apache.org/r/4115/ > > > ----------------------------------------------------------- > > > > > > (Updated 2012-03-01 00:31:40.789559) > > > > > > > > > Review request for Flume. > > > > > > > > > Changes > > > ------- > > > > > > Last diff included changes not intended for this jira. > > > > > > > > > Summary > > > ------- > > > > > > Currently the configuration of each component sits deep inside the > component themselves. There is no way to verify if a configuration is valid > before run time. In most systems like Flume, it is likely that these confs > will be deployed from one single host, on to the machines where flume > agents are running. Only when each agent starts, invalid confs are > identified because the Agents would terminate by throwing exceptions. This > is a first attempt to make a component-aware configuration system which is > independent of the Flume, and does not require the Flume jar to be > installed. Each component needs a configuration manager, which configures > the components. > > > > > > I have not completed the validation part yet, but have completed the > configuration managers for all the components that come in the flume jar. I > will add support for the other components, such as HDFS sink, JDBC channel > etc soon. I will also add the validation support to the classes, but would > like a review of the model being proposed here. > > > > > > > > > This addresses bug FLUME-992. > > > https://issues.apache.org/jira/browse/FLUME-992 > > > > > > > > > Diffs (updated) > > > ----- > > > > > > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java > bca0c50 > > > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java > e445d61 > > > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java > 6fbd6ef > > > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java > 1cf1c0c > > > > flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java > 7710d46 > > > > flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java > 68d95fb > > > flume-ng-configuration/pom.xml PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java > PRE-CREATION > > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java > PRE-CREATION > > > flume-ng-configuration/src/test/resources/flume-ng-error-test > PRE-CREATION > > > flume-ng-configuration/src/test/resources/flume-ng-test.conf > PRE-CREATION > > > flume-ng-core/pom.xml d753fa1 > > > flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85 > > > > flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java > 352bf08 > > > > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java > a7d5f94 > > > > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java > e79490e > > > > flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java > c63d0a1 > > > > flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java > 1df580e > > > > flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java > 8f22746 > > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java > 0fa4839 > > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java > 84492e5 > > > flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631 > > > > flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java > 257bab3 > > > > flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java > 9f5b856 > > > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java > 45c031d > > > flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java > 1adc5ff > > > > flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java > 859f4fd > > > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java > 7b079f9 > > > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java > a96016c > > > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java > d205bbc > > > flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8 > > > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java > 3392dff > > > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java > 8dad0b2 > > > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java > bc81f26 > > > flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java > 467785f > > > > flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java > 93ad3bf > > > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java > 92ff6fe > > > > flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java > 7e26e2a > > > > flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java > c5c3f2f > > > > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java > 6035270 > > > > flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java > c27f82c > > > > flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java > 579b257 > > > > flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java > b1e67f7 > > > > flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java > 6e3eb53 > > > > flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java > 5fe270a > > > > flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java > ddd9478 > > > flume-ng-node/pom.xml b9b062e > > > > flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java > d66f6d1 > > > > flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java > 97f72e1 > > > > flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java > 521b586 > > > > flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java > fb2a960 > > > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java > 3da90a5 > > > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java > 7d8ee8a > > > > flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java > 0a5498f > > > pom.xml d785762 > > > > > > Diff: https://reviews.apache.org/r/4115/diff > > > > > > > > > Testing > > > ------- > > > > > > All existing unit tests for the components whose configuration has now > been moved to these stubs pass. > > > > > > > > > Thanks, > > > > > > Hari > > -- Eric Sammer twitter: esammer data: www.cloudera.com
