Updated Branches: refs/heads/trunk d66bf94b1 -> afc806f2e
FLUME-2124. Upgrade Morphline Solr Sink to CDK 1.4.1 (Wolfgang Hoschek via Hari Shreedharan) Project: http://git-wip-us.apache.org/repos/asf/flume/repo Commit: http://git-wip-us.apache.org/repos/asf/flume/commit/afc806f2 Tree: http://git-wip-us.apache.org/repos/asf/flume/tree/afc806f2 Diff: http://git-wip-us.apache.org/repos/asf/flume/diff/afc806f2 Branch: refs/heads/trunk Commit: afc806f2eb01a78f9fa25dd38118df15ac6274d4 Parents: d66bf94 Author: Hari Shreedharan <[email protected]> Authored: Wed Jul 10 17:58:54 2013 -0700 Committer: Hari Shreedharan <[email protected]> Committed: Wed Jul 10 17:58:54 2013 -0700 ---------------------------------------------------------------------- flume-ng-doc/sphinx/FlumeUserGuide.rst | 4 ++-- flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml | 8 +++++++- 2 files changed, 9 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flume/blob/afc806f2/flume-ng-doc/sphinx/FlumeUserGuide.rst ---------------------------------------------------------------------- diff --git a/flume-ng-doc/sphinx/FlumeUserGuide.rst b/flume-ng-doc/sphinx/FlumeUserGuide.rst index 63cad21..fb42528 100644 --- a/flume-ng-doc/sphinx/FlumeUserGuide.rst +++ b/flume-ng-doc/sphinx/FlumeUserGuide.rst @@ -1835,7 +1835,7 @@ This sink extracts data from Flume events, transforms it, and loads it in near-r This sink is well suited for use cases that stream raw data into HDFS (via the HdfsSink) and simultaneously extract, transform and load the same data into Solr (via MorphlineSolrSink). In particular, this sink can process arbitrary heterogeneous raw data from disparate data sources and turn it into a data model that is useful to Search applications. -The ETL functionality is customizable using a `morphline configuration file <http://cloudera.github.io/cdk/docs/0.4.0/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe event records from one command to another. +The ETL functionality is customizable using a `morphline configuration file <http://cloudera.github.io/cdk/docs/0.4.1/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe event records from one command to another. Morphlines can be seen as an evolution of Unix pipelines where the data model is generalized to work with streams of generic records, including arbitrary binary payloads. A morphline command is a bit like a Flume Interceptor. Morphlines can be embedded into Hadoop components such as Flume. @@ -2595,7 +2595,7 @@ prefix "" The prefix string constant to prepend to each generat Morphline Interceptor ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This interceptor filters the events through a `morphline configuration file <http://cloudera.github.io/cdk/docs/0.4.0/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe records from one command to another. +This interceptor filters the events through a `morphline configuration file <http://cloudera.github.io/cdk/docs/0.4.1/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe records from one command to another. For example the morphline can ignore certain events or alter or insert certain event headers via regular expression based pattern matching, or it can auto-detect and set a MIME type via Apache Tika on events that are intercepted. For example, this kind of packet sniffing can be used for content based dynamic routing in a Flume topology. MorphlineInterceptor can also help to implement dynamic routing to multiple Apache Solr collections (e.g. for multi-tenancy). http://git-wip-us.apache.org/repos/asf/flume/blob/afc806f2/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml ---------------------------------------------------------------------- diff --git a/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml b/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml index 5115090..5452aa9 100644 --- a/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml +++ b/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml @@ -34,7 +34,7 @@ limitations under the License. <solr.version>4.3.0</solr.version> <solr.expected.version>4.3.0</solr.expected.version> <!-- sanity check to verify we actually run against the expected version rather than some outdated version --> <tika.version>1.3</tika.version> - <cdk.version>0.4.0</cdk.version> + <cdk.version>0.4.1</cdk.version> <slf4j.version>1.6.1</slf4j.version> <surefire.version>2.12.4</surefire.version> </properties> @@ -102,6 +102,12 @@ limitations under the License. <dependency> <groupId>com.cloudera.cdk</groupId> + <artifactId>cdk-morphlines-json</artifactId> + <version>${cdk.version}</version> + </dependency> + + <dependency> + <groupId>com.cloudera.cdk</groupId> <artifactId>cdk-morphlines-twitter</artifactId> <version>${cdk.version}</version> </dependency>
