Hi Eric, I had considered the validator pattern. What we are trying to do is to be able to validate the configuration for specific components specified in the conf, without actually depending on any of the Flume API themselves.
The use case I am trying to address: Say you have a 100 node cluster on which we want to install and run flume and we use some automated way of starting up flume on these clusters, usually from one box. The idea is that the conf file be verified before actually deploying it on the cluster itself. Requiring Flume to be installed on these boxes(which could as well be on the corp network) should not be a requirement. Using the validator pattern directly would require us to actually have the flume components on the box validating the conf. We could use a stub that stores all the required parameters and run the validator on that, but I think that is pretty much the same as what is already done here. Thanks! Hari -- Hari Shreedharan On Thursday, March 1, 2012 at 11:20 AM, Eric Sammer wrote: > Why not just follow the Validator pattern? See Spring's validators for > reference. > > > > On Feb 29, 2012, at 4:31 PM, Hari Shreedharan <[email protected] > (mailto:[email protected])> wrote: > > > > > ----------------------------------------------------------- > > This is an automatically generated e-mail. To reply, visit: > > https://reviews.apache.org/r/4115/ > > ----------------------------------------------------------- > > > > (Updated 2012-03-01 00:31:40.789559) > > > > > > Review request for Flume. > > > > > > Changes > > ------- > > > > Last diff included changes not intended for this jira. > > > > > > Summary > > ------- > > > > Currently the configuration of each component sits deep inside the > > component themselves. There is no way to verify if a configuration is valid > > before run time. In most systems like Flume, it is likely that these confs > > will be deployed from one single host, on to the machines where flume > > agents are running. Only when each agent starts, invalid confs are > > identified because the Agents would terminate by throwing exceptions. This > > is a first attempt to make a component-aware configuration system which is > > independent of the Flume, and does not require the Flume jar to be > > installed. Each component needs a configuration manager, which configures > > the components. > > > > I have not completed the validation part yet, but have completed the > > configuration managers for all the components that come in the flume jar. I > > will add support for the other components, such as HDFS sink, JDBC channel > > etc soon. I will also add the validation support to the classes, but would > > like a review of the model being proposed here. > > > > > > This addresses bug FLUME-992. > > https://issues.apache.org/jira/browse/FLUME-992 > > > > > > Diffs (updated) > > ----- > > > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java > > bca0c50 > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java > > e445d61 > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java > > 6fbd6ef > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java > > 1cf1c0c > > flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java > > 7710d46 > > flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java > > 68d95fb > > flume-ng-configuration/pom.xml PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java > > PRE-CREATION > > flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION > > flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION > > flume-ng-core/pom.xml d753fa1 > > flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85 > > flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java > > 352bf08 > > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java > > a7d5f94 > > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java > > e79490e > > flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java > > c63d0a1 > > flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java > > 1df580e > > flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java > > 8f22746 > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839 > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 > > flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631 > > flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java > > 257bab3 > > flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java > > 9f5b856 > > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java > > 45c031d > > flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff > > flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java > > 859f4fd > > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9 > > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c > > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java > > d205bbc > > flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8 > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java > > 3392dff > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java > > 8dad0b2 > > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java > > bc81f26 > > flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f > > flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java > > 93ad3bf > > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java > > 92ff6fe > > flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java > > 7e26e2a > > flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java > > c5c3f2f > > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java > > 6035270 > > flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java > > c27f82c > > flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java > > 579b257 > > flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java > > b1e67f7 > > flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java > > 6e3eb53 > > flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java > > 5fe270a > > flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java > > ddd9478 > > flume-ng-node/pom.xml b9b062e > > flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java > > d66f6d1 > > flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java > > 97f72e1 > > flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java > > 521b586 > > flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java > > fb2a960 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java > > 3da90a5 > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java > > 7d8ee8a > > flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java > > 0a5498f > > pom.xml d785762 > > > > Diff: https://reviews.apache.org/r/4115/diff > > > > > > Testing > > ------- > > > > All existing unit tests for the components whose configuration has now been > > moved to these stubs pass. > > > > > > Thanks, > > > > Hari
