Author: apaliwal
Date: Wed Jul 29 20:32:46 2015
New Revision: 1693341
URL: http://svn.apache.org/r1693341
Log:
Updated docs for FLUME-2660
Modified:
flume/site/trunk/content/sphinx/FlumeDeveloperGuide.rst
flume/site/trunk/content/sphinx/FlumeUserGuide.rst
Modified: flume/site/trunk/content/sphinx/FlumeDeveloperGuide.rst
URL:
http://svn.apache.org/viewvc/flume/site/trunk/content/sphinx/FlumeDeveloperGuide.rst?rev=1693341&r1=1693340&r2=1693341&view=diff
==============================================================================
--- flume/site/trunk/content/sphinx/FlumeDeveloperGuide.rst (original)
+++ flume/site/trunk/content/sphinx/FlumeDeveloperGuide.rst Wed Jul 29 20:32:46
2015
@@ -15,7 +15,7 @@
======================================
-Flume 1.6.0 Developer Guide
+Flume 1.7.0-SNAPSHOT Developer Guide
======================================
Introduction
@@ -644,9 +644,10 @@ Flume ``Transaction``.
:alt: Transaction sequence diagram
A ``Transaction`` is implemented within a ``Channel`` implementation. Each
-``Source`` and ``Sink`` that is connected to ``Channel`` must obtain a
-``Transaction`` object. The ``Source``\ s actually use a ``ChannelSelector``
-interface to encapsulate the ``Transaction``. The operation to stage an
+``Source`` and ``Sink`` that is connected to a ``Channel`` must obtain a
+``Transaction`` object. The ``Source``\ s use a ``ChannelProcessor``
+to manage the ``Transaction``\ s, the ``Sink``\ s manage them explicitly via
+their configured ``Channel``. The operation to stage an
``Event`` (put it into a ``Channel``) or extract an ``Event`` (take it out of a
``Channel``) is done inside an active ``Transaction``. For example:
@@ -759,8 +760,6 @@ processing its own configuration setting
if (t instanceof Error) {
throw (Error)t;
}
- } finally {
- txn.close();
}
return status;
}
@@ -770,13 +769,12 @@ Source
~~~~~~
The purpose of a ``Source`` is to receive data from an external client and
store
-it into the ``Channel``. A ``Source`` can get an instance of its own
-``ChannelProcessor`` to process an ``Event``. The ``ChannelProcessor`` in turn
-can get an instance of its own ``ChannelSelector`` that's used to get the
-``Channel``\ s associated with the ``Source``, as configured in the Flume
-properties file. A ``Transaction`` can then be retrieved from each associated
-``Channel`` so that the ``Source`` can place ``Event``\ s into the ``Channel``
-reliably, within a ``Transaction``.
+it into the configured ``Channel``\ s. A ``Source`` can get an instance of its
own
+``ChannelProcessor`` to process an ``Event``, commited within a ``Channel``
+local transaction, in serial. In the case of an exception, required
+``Channel``\ s will propagate the exception, all ``Channel``\ s will rollback
their
+transaction, but events processed previously on other ``Channel``\ s will
remain
+committed.
Similar to the ``SinkRunner.PollingRunner`` ``Runnable``, thereâs
a ``PollingRunner`` ``Runnable`` that executes on a thread created when the
@@ -826,24 +824,17 @@ mechanism that captures the new data and
public Status process() throws EventDeliveryException {
Status status = null;
- // Start transaction
- Channel ch = getChannel();
- Transaction txn = ch.getTransaction();
- txn.begin();
try {
- // This try clause includes whatever Channel operations you want to do
+ // This try clause includes whatever Channel/Event operations you want
to do
// Receive new data
Event e = getSomeData();
// Store the Event into this Source's associated Channel(s)
- getChannelProcessor().processEvent(e)
+ getChannelProcessor().processEvent(e);
- txn.commit();
status = Status.READY;
} catch (Throwable t) {
- txn.rollback();
-
// Log exception, handle individual exceptions as needed
status = Status.BACKOFF;
Modified: flume/site/trunk/content/sphinx/FlumeUserGuide.rst
URL:
http://svn.apache.org/viewvc/flume/site/trunk/content/sphinx/FlumeUserGuide.rst?rev=1693341&r1=1693340&r2=1693341&view=diff
==============================================================================
--- flume/site/trunk/content/sphinx/FlumeUserGuide.rst (original)
+++ flume/site/trunk/content/sphinx/FlumeUserGuide.rst Wed Jul 29 20:32:46 2015
@@ -15,7 +15,7 @@
======================================
-Flume 1.6.0 User Guide
+Flume 1.7.0-SNAPSHOT User Guide
======================================
Introduction
@@ -1152,6 +1152,13 @@ Property Name Default
batchSize 1000 Maximum number of messages
written to Channel in one batch
batchDurationMillis 1000 Maximum time (in ms) before a
batch will be written to Channel
The batch will be written
whenever the first of size and time will be reached.
+backoffSleepIncrement 1000 Initial and incremental wait
time that is triggered when a Kafka Topic appears to be empty.
+ Wait period will reduce
aggressive pinging of an empty Kafka Topic. One second is ideal for
+ ingestion use cases but a lower
value may be required for low latency operations with
+ interceptors.
+maxBackoffSleep 5000 Maximum wait time that is
triggered when a Kafka Topic appears to be empty. Five seconds is
+ ideal for ingestion use cases
but a lower value may be required for low latency operations
+ with interceptors.
Other Kafka Consumer Properties -- These properties are used to
configure the Kafka Consumer. Any producer property supported
by Kafka can be used. The only
requirement is to prepend the property name with the prefix ``kafka.``.
For example:
kafka.consumer.timeout.ms
@@ -1952,7 +1959,7 @@ trust-all-certs false
truststore --
The path to a custom Java truststore file. Flume uses the certificate
authority information in this file to determine whether the remote Avro
Source's SSL authentication credentials should be trusted. If not specified,
the default Java JSSE certificate authority files (typically "jssecacerts" or
"cacerts" in the Oracle JRE) will be used.
truststore-password --
The password for the specified truststore.
truststore-type JKS
The type of the Java truststore. This can be "JKS" or other supported Java
truststore type.
-exclude-protocols SSLv2Hello SSLv3
Space-separated list of SSL/TLS protocols to exclude
+exclude-protocols SSLv3
Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be
excluded in addition to the protocols specified.
maxIoWorkers 2 * the number of available processors in the
machine The maximum number of I/O worker threads. This is configured on the
NettyAvroRpcClient NioClientSocketChannelFactory.
==========================
=====================================================
===========================================================================================
@@ -2241,17 +2248,20 @@ The type is the FQCN: org.apache.flume.s
Required properties are in **bold**.
-===================
=======================================================================
========================
-Property Name Default
Description
-===================
=======================================================================
========================
-**channel** --
-**type** --
The component type name, needs to be
``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
-**morphlineFile** --
The relative or absolute path on the local file system to the
morphline configuration file. Example: ``/etc/flume-ng/conf/morphline.conf``
-morphlineId null
Optional name used to identify a morphline if there are multiple
morphlines in a morphline config file
-batchSize 1000
The maximum number of events to take per flume transaction.
-batchDurationMillis 1000
The maximum duration per flume transaction (ms). The transaction
commits after this duration or when batchSize is exceeded, whichever comes
first.
-handlerClass org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl
The FQCN of a class implementing
org.apache.flume.sink.solr.morphline.MorphlineHandler
-===================
=======================================================================
========================
+===============================
=======================================================================
========================
+Property Name Default
Description
+===============================
=======================================================================
========================
+**channel** --
+**type** --
The component type name, needs to be
``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
+**morphlineFile** --
The relative or absolute path on the local file
system to the morphline configuration file. Example:
``/etc/flume-ng/conf/morphline.conf``
+morphlineId null
Optional name used to identify a morphline if there
are multiple morphlines in a morphline config file
+batchSize 1000
The maximum number of events to take per flume
transaction.
+batchDurationMillis 1000
The maximum duration per flume transaction (ms).
The transaction commits after this duration or when batchSize is exceeded,
whichever comes first.
+handlerClass
org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl The
FQCN of a class implementing
org.apache.flume.sink.solr.morphline.MorphlineHandler
+isProductionMode false
This flag should be enabled for mission critical,
large-scale online production systems that need to make progress without
downtime when unrecoverable exceptions occur. Corrupt or malformed parser input
data, parser bugs, and errors related to unknown Solr schema fields produce
unrecoverable exceptions.
+recoverableExceptionClasses
org.apache.solr.client.solrj.SolrServerException Comma
separated list of recoverable exceptions that tend to be transient, in which
case the corresponding task can be retried. Examples include network connection
errors, timeouts, etc. When the production mode flag is set to true, the
recoverable exceptions configured using this parameter will not be ignored and
hence will lead to retries.
+isIgnoringRecoverableExceptions false
This flag should be enabled, if an unrecoverable
exception is accidentally misclassified as recoverable. This enables the sink
to make progress and avoid retrying an event forever.
+===============================
=======================================================================
========================
Example for agent named a1:
@@ -3420,7 +3430,7 @@ Log4J Appender
Appends Log4j events to a flume agent's avro source. A client using this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar).
+flume-ng-sdk-1.7.0-SNAPSHOT.jar).
Required properties are in **bold**.
===================== =======
==================================================================================
@@ -3484,7 +3494,7 @@ Load Balancing Log4J Appender
Appends Log4j events to a list of flume agent's avro source. A client using
this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar). This appender supports a round-robin and random
+flume-ng-sdk-1.7.0-SNAPSHOT.jar). This appender supports a round-robin and
random
scheme for performing the load balancing. It also supports a configurable
backoff
timeout so that down agents are removed temporarily from the set of hosts
Required properties are in **bold**.
