http://git-wip-us.apache.org/repos/asf/incubator-samza/blob/1e2cfe22/docs/learn/documentation/0.7.0/jobs/configuration-table.html ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/0.7.0/jobs/configuration-table.html b/docs/learn/documentation/0.7.0/jobs/configuration-table.html deleted file mode 100644 index ea6b1ef..0000000 --- a/docs/learn/documentation/0.7.0/jobs/configuration-table.html +++ /dev/null @@ -1,1157 +0,0 @@ -<!DOCTYPE html> -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> -<html> - <head> - <meta charset="utf-8"> - <title>Samza Configuration Reference</title> - <style type="text/css"> - body { - font-family: "Helvetica Neue",Helvetica,Arial,sans-serif; - font-size: 14px; - line-height: 22px; - color: #333; - background-color: #fff; - } - - table { - border-collapse: collapse; - margin: 1em 0; - } - - table th, table td { - text-align: left; - vertical-align: top; - padding: 12px; - border-bottom: 1px solid #ccc; - border-top: 1px solid #ccc; - border-left: 0; - border-right: 0; - } - - table td.property, table td.default { - white-space: nowrap; - } - - table th.section { - background-color: #eee; - } - - table th.section .subtitle { - font-weight: normal; - } - - code, a.property { - font-family: monospace; - } - - span.system, span.stream, span.store, span.serde, span.rewriter, span.listener, span.reporter { - padding: 1px; - margin: 1px; - border-width: 1px; - border-style: solid; - border-radius: 4px; - } - - span.system { - background-color: #ddf; - border-color: #bbd; - } - - span.stream { - background-color: #dfd; - border-color: #bdb; - } - - span.store { - background-color: #fdf; - border-color: #dbd; - } - - span.serde { - background-color: #fdd; - border-color: #dbb; - } - - span.rewriter { - background-color: #eee; - border-color: #ccc; - } - - span.listener { - background-color: #ffd; - border-color: #ddb; - } - - span.reporter { - background-color: #dff; - border-color: #bdd; - } - </style> - </head> - - <body> - <h1>Samza Configuration Reference</h1> - <p>The following table lists all the standard properties that can be included in a Samza job configuration file.</p> - <p>Words highlighted like <span class="system">this</span> are placeholders for your own variable names.</p> - <table> - <tbody> - <tr><th>Name</th><th>Default</th><th>Description</th></tr> - <tr> - <th colspan="3" class="section" id="job"><a href="configuration.html">Samza job configuration</a></th> - </tr> - - <tr> - <td class="property" id="job-factory-class">job.factory.class</td> - <td class="default"></td> - <td class="description"> - <strong>Required:</strong> The <a href="job-runner.html">job factory</a> to use for running this job. - The value is a fully-qualified Java classname, which must implement - <a href="../api/javadocs/org/apache/samza/job/StreamJobFactory.html">StreamJobFactory</a>. - Samza ships with two implementations: - <dl> - <dt><code>org.apache.samza.job.local.ThreadJobFactory</code></dt> - <dd>Runs your job on your local machine using threads. This is intended only for - development, not for production deployments.</dd> - <dt><code>org.apache.samza.job.local.ProcessJobFactory</code></dt> - <dd>Runs your job on your local machine as a subprocess. An optional command builder - property can also be specified (see <a href="#task-command-class" class="property"> - task.command.class</a> for details). This is intended only for development, - not for production deployments.</dd> - <dt><code>org.apache.samza.job.yarn.YarnJobFactory</code></dt> - <dd>Runs your job on a YARN grid. See <a href="#yarn">below</a> for YARN-specific configuration.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="job-name">job.name</td> - <td class="default"></td> - <td class="description"> - <strong>Required:</strong> The name of your job. This name appears on the Samza dashboard, and it - is used to tell apart this job's checkpoints from other jobs' checkpoints. - </td> - </tr> - - <tr> - <td class="property" id="job-id">job.id</td> - <td class="default">1</td> - <td class="description"> - If you run several instances of your job at the same time, you need to give each execution a - different <code>job.id</code>. This is important, since otherwise the jobs will overwrite each - others' checkpoints, and perhaps interfere with each other in other ways. - </td> - </tr> - - <tr> - <td class="property" id="job-config-rewriter-class">job.config.rewriter.<br><span class="rewriter">rewriter-name</span>.class</td> - <td class="default"></td> - <td class="description"> - You can optionally define configuration rewriters, which have the opportunity to dynamically - modify the job configuration before the job is started. For example, this can be useful for - pulling configuration from an external configuration management system, or for determining - the set of input streams dynamically at runtime. The value of this property is a - fully-qualified Java classname which must implement - <a href="../api/javadocs/org/apache/samza/config/ConfigRewriter.html">ConfigRewriter</a>. - Samza ships with one rewriter by default: - <dl> - <dt><code>org.apache.samza.config.RegExTopicGenerator</code></dt> - <dd>When consuming from Kafka, this allows you to consume all Kafka topics that match - some regular expression (rather than having to list each topic explicitly). - This rewriter has <a href="#regex-rewriter">additional configuration</a>.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="job-config-rewriters">job.config.rewriters</td> - <td class="default"></td> - <td class="description"> - If you have defined configuration rewriters, you need to list them here, in the order in - which they should be applied. The value of this property is a comma-separated list of - <span class="rewriter">rewriter-name</span> tokens. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="task"><a href="../api/overview.html">Task configuration</a></th> - </tr> - - <tr> - <td class="property" id="task-class">task.class</td> - <td class="default"></td> - <td class="description"> - <strong>Required:</strong> The fully-qualified name of the Java class which processes - incoming messages from input streams. The class must implement - <a href="../api/javadocs/org/apache/samza/task/StreamTask.html">StreamTask</a>, and may optionally implement - <a href="../api/javadocs/org/apache/samza/task/InitableTask.html">InitableTask</a>, - <a href="../api/javadocs/org/apache/samza/task/ClosableTask.html">ClosableTask</a> and/or - <a href="../api/javadocs/org/apache/samza/task/WindowableTask.html">WindowableTask</a>. - The class will be instantiated several times, once for every - <a href="../container/samza-container.html#tasks-and-partitions">input stream partition</a>. - </td> - </tr> - - <tr> - <td class="property" id="task-inputs">task.inputs</td> - <td class="default"></td> - <td class="description"> - <strong>Required:</strong> A comma-separated list of streams that are consumed by this job. - Each stream is given in the format - <span class="system">system-name</span>.<span class="stream">stream-name</span>. - For example, if you have one input system called <code>my-kafka</code>, and want to consume two - Kafka topics called <code>PageViewEvent</code> and <code>UserActivityEvent</code>, then you would set - <code>task.inputs=my-kafka.PageViewEvent, my-kafka.UserActivityEvent</code>. - </td> - </tr> - - <tr> - <td class="property" id="task-window-ms">task.window.ms</td> - <td class="default">-1</td> - <td class="description"> - If <a href="#task-class" class="property">task.class</a> implements - <a href="../api/javadocs/org/apache/samza/task/WindowableTask.html">WindowableTask</a>, it can - receive a <a href="../container/windowing.html">windowing callback</a> in regular intervals. - This property specifies the time between window() calls, in milliseconds. If the number is - negative (the default), window() is never called. Note that Samza is - <a href="../container/event-loop.html">single-threaded</a>, so a window() call will never - occur concurrently with the processing of a message. If a message is being processed at the - time when a window() call is due, the window() call occurs after the processing of the current - message has completed. - </td> - </tr> - - <tr> - <td class="property" id="task-checkpoint-factory">task.checkpoint.factory</td> - <td class="default"></td> - <td class="description"> - To enable <a href="../container/checkpointing.html">checkpointing</a>, you must set - this property to the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/checkpoint/CheckpointManagerFactory.html">CheckpointManagerFactory</a>. - This is not required, but recommended for most jobs. If you don't configure checkpointing, - and a job or container restarts, it does not remember which messages it has already processed. - Without checkpointing, consumer behavior is determined by the - <a href="#systems-samza-offset-default" class="property">...samza.offset.default</a> - setting, which by default skips any messages that were published while the container was - restarting. Checkpointing allows a job to start up where it previously left off. - Samza ships with two checkpoint managers by default: - <dl> - <dt><code>org.apache.samza.checkpoint.file.FileSystemCheckpointManagerFactory</code></dt> - <dd>Writes checkpoints to files on the local filesystem. You can configure the file path - with the <a href="#task-checkpoint-path" class="property">task.checkpoint.path</a> - property. This is a simple option if your job always runs on the same machine. - On a multi-machine cluster, this would require a network filesystem mount.</dd> - <dt><code>org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory</code></dt> - <dd>Writes checkpoints to a dedicated topic on a Kafka cluster. This is the recommended - option if you are already using Kafka for input or output streams. Use the - <a href="#task-checkpoint-system" class="property">task.checkpoint.system</a> - property to configure which Kafka cluster to use for checkpoints.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="task-commit-ms">task.commit.ms</td> - <td class="default">60000</td> - <td class="description"> - If <a href="#task-checkpoint-factory" class="property">task.checkpoint.factory</a> is - configured, this property determines how often a checkpoint is written. The value is - the time between checkpoints, in milliseconds. The frequency of checkpointing affects - failure recovery: if a container fails unexpectedly (e.g. due to crash or machine failure) - and is restarted, it resumes processing at the last checkpoint. Any messages processed - since the last checkpoint on the failed container are processed again. Checkpointing - more frequently reduces the number of messages that may be processed twice, but also - uses more resources. - </td> - </tr> - - <tr> - <td class="property" id="task-command-class">task.command.class</td> - <td class="default">org.apache.samza.job.<br>ShellCommandBuilder</td> - <td class="description"> - The fully-qualified name of the Java class which determines the command line and environment - variables for a <a href="../container/samza-container.html">container</a>. It must be a subclass of - <a href="../api/javadocs/org/apache/samza/job/CommandBuilder.html">CommandBuilder</a>. - This defaults to <code>task.command.class=org.apache.samza.job.ShellCommandBuilder</code>. - </td> - </tr> - - <tr> - <td class="property" id="task-opts">task.opts</td> - <td class="default"></td> - <td class="description"> - Any JVM options to include in the command line when executing Samza containers. For example, - this can be used to set the JVM heap size, to tune the garbage collector, or to enable - <a href="/learn/tutorials/0.7.0/remote-debugging-samza.html">remote debugging</a>. Note - there are some issues with the current implementation of <code>task.opts</code>: - <ul> - <li>If you set this property, the log configuration is disrupted. Please see - <a href="https://issues.apache.org/jira/browse/SAMZA-109";>SAMZA-109</a> for a workaround.</li> - <li>This cannot be used when running with <code>ThreadJobFactory</code></li> - </ul> - </td> - </tr> - - <tr> - <td class="property" id="task-execute">task.execute</td> - <td class="default">bin/run-container.sh</td> - <td class="description"> - The command that starts a Samza container. The script must be included in the - <a href="packaging.html">job package</a>. There is usually no need to customize this. - </td> - </tr> - - <tr> - <td class="property" id="task-chooser-class">task.chooser.class</td> - <td class="default">org.apache.samza.<br>system.chooser.<br>RoundRobinChooserFactory</td> - <td class="description"> - This property can be optionally set to override the default - <a href="../container/streams.html#messagechooser">message chooser</a>, which determines the - order in which messages from multiple input streams are processed. The value of this - property is the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/system/chooser/MessageChooserFactory.html">MessageChooserFactory</a>. - </td> - </tr> - - <tr> - <td class="property" id="task-lifecycle-listener-class">task.lifecycle.listener.<br><span class="listener">listener-name</span>.class</td> - <td class="default"></td> - <td class="description"> - Use this property to register a - <a href="../container/event-loop.html#lifecycle-listeners">lifecycle listener</a>, which can receive - a notification when a container starts up or shuts down, or when a message is processed. - The value is the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/task/TaskLifecycleListenerFactory.html">TaskLifecycleListenerFactory</a>. - You can define multiple lifecycle listeners, each with a different <span class="listener">listener-name</span>, - and reference them in <a href="#task-lifecycle-listeners" class="property">task.lifecycle.listeners</a>. - </td> - </tr> - - <tr> - <td class="property" id="task-lifecycle-listeners">task.lifecycle.listeners</td> - <td class="default"></td> - <td class="description"> - If you have defined <a href="../container/event-loop.html#lifecycle-listeners">lifecycle listeners</a> with - <a href="#task-lifecycle-listener-class" class="property">task.lifecycle.listener.*.class</a>, - you need to list them here in order to enable them. The value of this property is a - comma-separated list of <span class="listener">listener-name</span> tokens. - </td> - </tr> - - <tr> - <td class="property" id="task-drop-deserialization-errors">task.drop.deserialization.errors</td> - <td class="default"></td> - <td class="description"> - This property is to define how the system deals with deserialization failure situation. If set to true, the system will - skip the error messages and keep running. If set to false, the system with throw exceptions and fail the container. Default - is false. - </td> - </tr> - - <tr> - <td class="property" id="task-drop-serialization-errors">task.drop.serialization.errors</td> - <td class="default"></td> - <td class="description"> - This property is to define how the system deals with serialization failure situation. If set to true, the system will - drop the error messages and keep running. If set to false, the system with throw exceptions and fail the container. Default - is false. - </td> - </tr> - - <tr> - <td class="property" id="task-poll-interval-ms">task.poll.interval.ms</td> - <td class="default"></td> - <td class="description"> - Samza's container polls for more messages under two conditions. The first condition arises when there are simply no remaining - buffered messages to process for any input SystemStreamPartition. The second condition arises when some input - SystemStreamPartitions have empty buffers, but some do not. In the latter case, a polling interval is defined to determine how - often to refresh the empty SystemStreamPartition buffers. By default, this interval is 50ms, which means that any empty - SystemStreamPartition buffer will be refreshed at least every 50ms. A higher value here means that empty SystemStreamPartitions - will be refreshed less often, which means more latency is introduced, but less CPU and network will be used. Decreasing this - value means that empty SystemStreamPartitions are refreshed more frequently, thereby introducing less latency, but increasing - CPU and network utilization. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="streams"><a href="../container/streams.html">Systems (input and output streams)</a></th> - </tr> - - <tr> - <td class="property" id="systems-samza-factory">systems.<span class="system">system-name</span>.<br>samza.factory</td> - <td class="default"></td> - <td class="description"> - <strong>Required:</strong> The fully-qualified name of a Java class which provides a - <em>system</em>. A system can provide input streams which you can consume in your Samza job, - or output streams to which you can write, or both. The requirements on a system are very - flexible — it may connect to a message broker, or read and write files, or use a database, - or anything else. The class must implement - <a href="../api/javadocs/org/apache/samza/system/SystemFactory.html">SystemFactory</a>. - Samza ships with the following implementations: - <dl> - <dt><code>org.apache.samza.system.kafka.KafkaSystemFactory</code></dt> - <dd>Connects to a cluster of <a href="http://kafka.apache.org/";>Kafka</a> brokers, allows - Kafka topics to be consumed as streams in Samza, allows messages to be published to - Kafka topics, and allows Kafka to be used for checkpointing (see - <a href="#task-checkpoint-factory" class="property">task.checkpoint.factory</a>). - See also <a href="#kafka">configuration of a Kafka system</a>.</dd> - <dt><code>org.apache.samza.system.filereader.FileReaderSystemFactory</code></dt> - <dd>Reads data from a file on the local filesystem (the stream name is the path of the - file to read). The file is read as ASCII, and treated as a stream of messages separated - by newline (<code>\n</code>) characters. A task can consume each line of the file as - a <code>java.lang.String</code> object. This system does not provide output streams.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-key-serde">systems.<span class="system">system-name</span>.<br>samza.key.serde</td> - <td class="default" rowspan="2"></td> - <td class="description" rowspan="2"> - The <a href="../container/serialization.html">serde</a> which will be used to deserialize the - <em>key</em> of messages on input streams, and to serialize the <em>key</em> of messages on - output streams. This property can be defined either for an individual stream, or for all - streams within a system (if both are defined, the stream-level definition takes precedence). - The value of this property must be a <span class="serde">serde-name</span> that is registered - with <a href="#serializers-registry-class" class="property">serializers.registry.*.class</a>. - If this property is not set, messages are passed unmodified between the input stream consumer, - the task and the output stream producer. - </td> - </tr> - <tr> - <td class="property">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.key.serde</td> - </tr> - - <tr> - <td class="property" id="systems-samza-msg-serde">systems.<span class="system">system-name</span>.<br>samza.msg.serde</td> - <td class="default" rowspan="2"></td> - <td class="description" rowspan="2"> - The <a href="../container/serialization.html">serde</a> which will be used to deserialize the - <em>value</em> of messages on input streams, and to serialize the <em>value</em> of messages on - output streams. This property can be defined either for an individual stream, or for all - streams within a system (if both are defined, the stream-level definition takes precedence). - The value of this property must be a <span class="serde">serde-name</span> that is registered - with <a href="#serializers-registry-class" class="property">serializers.registry.*.class</a>. - If this property is not set, messages are passed unmodified between the input stream consumer, - the task and the output stream producer. - </td> - </tr> - <tr> - <td class="property">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.msg.serde</td> - </tr> - - <tr> - <td class="property" id="systems-samza-offset-default">systems.<span class="system">system-name</span>.<br>samza.offset.default</td> - <td class="default" rowspan="2">upcoming</td> - <td class="description" rowspan="2"> - If a container starts up without a <a href="../container/checkpointing.html">checkpoint</a>, - this property determines where in the input stream we should start consuming. The value must be an - <a href="../api/javadocs/org/apache/samza/system/SystemStreamMetadata.OffsetType.html">OffsetType</a>, - one of the following: - <dl> - <dt><code>upcoming</code></dt> - <dd>Start processing messages that are published after the job starts. Any messages published while - the job was not running are not processed.</dd> - <dt><code>oldest</code></dt> - <dd>Start processing at the oldest available message in the system, and - <a href="reprocessing.html">reprocess</a> the entire available message history.</dd> - </dl> - This property can be defined either for an individual stream, or for all streams within a system - (if both are defined, the stream-level definition takes precedence). - </td> - </tr> - <tr> - <td class="property">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.offset.default</td> - </tr> - - <tr> - <td class="property" id="systems-streams-samza-reset-offset">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.reset.offset</td> - <td>false</td> - <td> - If set to <code>true</code>, when a Samza container starts up, it ignores any - <a href="../container/checkpointing.html">checkpointed offset</a> for this particular input - stream. Its behavior is thus determined by the <code>samza.offset.default</code> setting. - Note that the reset takes effect <em>every time a container is started</em>, which may be - every time you restart your job, or more frequently if a container fails and is restarted - by the framework. - </td> - </tr> - - <tr> - <td class="property" id="systems-streams-samza-priority">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.priority</td> - <td>-1</td> - <td> - If one or more streams have a priority set (any positive integer), they will be processed - with <a href="../container/streams.html#prioritizing-input-streams">higher priority</a> than the other streams. - You can set several streams to the same priority, or define multiple priority levels by - assigning a higher number to the higher-priority streams. If a higher-priority stream has - any messages available, they will always be processed first; messages from lower-priority - streams are only processed when there are no new messages on higher-priority inputs. - </td> - </tr> - - <tr> - <td class="property" id="systems-streams-samza-bootstrap">systems.<span class="system">system-name</span>.<br>streams.<span class="stream">stream-name</span>.<br>samza.bootstrap</td> - <td>false</td> - <td> - If set to <code>true</code>, this stream will be processed as a - <a href="../container/streams.html#bootstrapping">bootstrap stream</a>. This means that every time - a Samza container starts up, this stream will be fully consumed before messages from any - other stream are processed. - </td> - </tr> - - <tr> - <td class="property" id="task-consumer-batch-size">task.consumer.batch.size</td> - <td>1</td> - <td> - If set to a positive integer, the task will try to consume - <a href="../container/streams.html#batching">batches</a> with the given number of messages - from each input stream, rather than consuming round-robin from all the input streams on - each individual message. Setting this property can improve performance in some cases. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="serdes"><a href="../container/serialization.html">Serializers/Deserializers (Serdes)</a></th> - </tr> - - <tr> - <td class="property" id="serializers-registry-class">serializers.registry.<br><span class="serde">serde-name</span>.class</td> - <td class="default"></td> - <td class="description"> - Use this property to register a <a href="../container/serialization.html">serializer/deserializer</a>, - which defines a way of encoding application objects as an array of bytes (used for messages - in streams, and for data in persistent storage). You can give a serde any - <span class="serde">serde-name</span> you want, and reference that name in properties like - <a href="#systems-samza-key-serde" class="property">systems.*.samza.key.serde</a>, - <a href="#systems-samza-msg-serde" class="property">systems.*.samza.msg.serde</a>, - <a href="#stores-key-serde" class="property">stores.*.key.serde</a> and - <a href="#stores-msg-serde" class="property">stores.*.msg.serde</a>. - The value of this property is the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/serializers/SerdeFactory.html">SerdeFactory</a>. - Samza ships with several serdes: - <dl> - <dt><code>org.apache.samza.serializers.ByteSerdeFactory</code></dt> - <dd>A no-op serde which passes through the undecoded byte array.</dd> - <dt><code>org.apache.samza.serializers.IntegerSerdeFactory</code></dt> - <dd>Encodes <code>java.lang.Integer</code> objects as binary (4 bytes fixed-length big-endian encoding).</dd> - <dt><code>org.apache.samza.serializers.StringSerdeFactory</code></dt> - <dd>Encodes <code>java.lang.String</code> objects as UTF-8.</dd> - <dt><code>org.apache.samza.serializers.JsonSerdeFactory</code></dt> - <dd>Encodes nested structures of <code>java.util.Map</code>, <code>java.util.List</code> etc. as JSON.</dd> - <dt><code>org.apache.samza.serializers.MetricsSnapshotSerdeFactory</code></dt> - <dd>Encodes <code>org.apache.samza.metrics.reporter.MetricsSnapshot</code> objects (which are - used for <a href="../container/metrics.html">reporting metrics</a>) as JSON.</dd> - <dt><code>org.apache.samza.serializers.KafkaSerdeFactory</code></dt> - <dd>Adapter which allows existing <code>kafka.serializer.Encoder</code> and - <code>kafka.serializer.Decoder</code> implementations to be used as Samza serdes. - Set serializers.registry.<span class="serde">serde-name</span>.encoder and - serializers.registry.<span class="serde">serde-name</span>.decoder to the appropriate - class names.</dd> - </dl> - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="filesystem-checkpoints"> - Using the filesystem for checkpoints<br> - <span class="subtitle"> - (This section applies if you have set - <a href="#task-checkpoint-factory" class="property">task.checkpoint.factory</a> - <code>= org.apache.samza.checkpoint.file.FileSystemCheckpointManagerFactory</code>) - </span> - </th> - </tr> - - <tr> - <td class="property" id="task-checkpoint-path">task.checkpoint.path</td> - <td class="default"></td> - <td class="description"> - Required if you are using the filesystem for checkpoints. Set this to the path on your local filesystem - where checkpoint files should be stored. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="kafka"> - Using <a href="http://kafka.apache.org/";>Kafka</a> for input streams, output streams and checkpoints<br> - <span class="subtitle"> - (This section applies if you have set - <a href="#systems-samza-factory" class="property">systems.*.samza.factory</a> - <code>= org.apache.samza.system.kafka.KafkaSystemFactory</code>) - </span> - </th> - </tr> - - <tr> - <td class="property" id="systems-samza-consumer-zookeeper-connect">systems.<span class="system">system-name</span>.<br>consumer.zookeeper.connect</td> - <td class="default"></td> - <td class="description"> - The hostname and port of one or more Zookeeper nodes where information about the - Kafka cluster can be found. This is given as a comma-separated list of - <code>hostname:port</code> pairs, such as - <code>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</code>. - If the cluster information is at some sub-path of the Zookeeper namespace, you need to - include the path at the end of the list of hostnames, for example: - <code>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181/clusters/my-kafka</code> - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-consumer-auto-offset-reset">systems.<span class="system">system-name</span>.<br>consumer.auto.offset.reset</td> - <td class="default">largest</td> - <td class="description"> - This setting determines what happens if a consumer attempts to read an offset that is - outside of the current valid range. This could happen if the topic does not exist, or - if a checkpoint is older than the maximum message history retained by the brokers. - This property is not to be confused with - <a href="#systems-samza-offset-default">systems.*.samza.offset.default</a>, - which determines what happens if there is no checkpoint. The following are valid - values for <code>auto.offset.reset</code>: - <dl> - <dt><code>smallest</code></dt> - <dd>Start consuming at the smallest (oldest) offset available on the broker - (process as much message history as available).</dd> - <dt><code>largest</code></dt> - <dd>Start consuming at the largest (newest) offset available on the broker - (skip any messages published while the job was not running).</dd> - <dt>anything else</dt> - <dd>Throw an exception and refuse to start up the job.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-consumer">systems.<span class="system">system-name</span>.<br>consumer.*</td> - <td class="default"></td> - <td class="description"> - Any <a href="http://kafka.apache.org/documentation.html#consumerconfigs";>Kafka consumer configuration</a> - can be included here. For example, to change the socket timeout, you can set - systems.<span class="system">system-name</span>.consumer.socket.timeout.ms. - (There is no need to configure <code>group.id</code> or <code>client.id</code>, - as they are automatically configured by Samza. Also, there is no need to set - <code>auto.commit.enable</code> because Samza has its own checkpointing mechanism.) - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-producer-metadata-broker-list">systems.<span class="system">system-name</span>.<br>producer.metadata.broker.list</td> - <td class="default"></td> - <td class="description"> - A list of network endpoints where the Kafka brokers are running. This is given as - a comma-separated list of <code>hostname:port</code> pairs, for example - <code>kafka1.example.com:9092,kafka2.example.com:9092,kafka3.example.com:9092</code>. - It's not necessary to list every single Kafka node in the cluster: Samza uses this - property in order to discover which topics and partitions are hosted on which broker. - This property is needed even if you are only consuming from Kafka, and not writing - to it, because Samza uses it to discover metadata about streams being consumed. - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-producer-producer-type">systems.<span class="system">system-name</span>.<br>producer.producer.type</td> - <td class="default">sync</td> - <td class="description"> - Controls whether messages emitted from a stream processor should be buffered before - they are sent to Kafka. The options are: - <dl> - <dt><code>sync</code></dt> - <dd>Any messages sent to output streams are synchronously flushed to the Kafka brokers - before the next message from an input stream is processed.</dd> - <dt><code>async</code></dt> - <dd>Messages sent to output streams are buffered within the Samza container, and published - to the Kafka brokers as a batch. This setting can increase throughput, but - risks buffered messages being lost if a container abruptly fails. The maximum - number of messages to buffer is controlled with - systems.<span class="system">system-name</span>.producer.batch.num.messages - and the maximum time (in milliseconds) to wait before flushing the buffer is set with - systems.<span class="system">system-name</span>.producer.queue.buffering.max.ms.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-producer">systems.<span class="system">system-name</span>.<br>producer.*</td> - <td class="default"></td> - <td class="description"> - Any <a href="http://kafka.apache.org/documentation.html#producerconfigs";>Kafka producer configuration</a> - can be included here. For example, to change the request timeout, you can set - systems.<span class="system">system-name</span>.producer.request.timeout.ms. - (There is no need to configure <code>client.id</code> as it is automatically - configured by Samza.) - </td> - </tr> - - <tr> - <td class="property" id="systems-samza-fetch-threshold">systems.<span class="system">system-name</span>.<br>samza.fetch.threshold</td> - <td class="default">50000</td> - <td class="description"> - When consuming streams from Kafka, a Samza container maintains an in-memory buffer - for incoming messages in order to increase throughput (the stream task can continue - processing buffered messages while new messages are fetched from Kafka). This - parameter determines the number of messages we aim to buffer across all stream - partitions consumed by a container. For example, if a container consumes 50 partitions, - it will try to buffer 1000 messages per partition by default. When the number of - buffered messages falls below that threshold, Samza fetches more messages from the - Kafka broker to replenish the buffer. Increasing this parameter can increase a job's - processing throughput, but also increases the amount of memory used. - </td> - </tr> - - <tr> - <td class="property" id="task-checkpoint-system">task.checkpoint.system</td> - <td class="default"></td> - <td class="description"> - This property is required if you are using Kafka for checkpoints - (<a href="#task-checkpoint-factory" class="property">task.checkpoint.factory</a> - <code>= org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory</code>). - You must set it to the <span class="system">system-name</span> of a Kafka system. The stream - name (topic name) within that system is automatically determined from the job name and ID: - <code>__samza_checkpoint_${<a href="#job-name" class="property">job.name</a>}_${<a href="#job-id" class="property">job.id</a>}</code> - (with underscores in the job name and ID replaced by hyphens). - </td> - </tr> - - <tr> - <td class="property" id="task-checkpoint-replication-factor">task.checkpoint.<br>replication.factor</td> - <td class="default">3</td> - <td class="description"> - If you are using Kafka for checkpoints, this is the number of Kafka nodes to which you want the - checkpoint topic replicated for durability. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="regex-rewriter"> - Consuming all Kafka topics matching a regular expression<br> - <span class="subtitle"> - (This section applies if you have set - <a href="#job-config-rewriter-class" class="property">job.config.rewriter.*.class</a> - <code>= org.apache.samza.config.RegExTopicGenerator</code>) - </span> - </th> - </tr> - - <tr> - <td class="property" id="job-config-rewriter-system">job.config.rewriter.<br><span class="rewriter">rewriter-name</span>.system</td> - <td class="default"></td> - <td class="description"> - Set this property to the <span class="system">system-name</span> of the Kafka system - from which you want to consume all matching topics. - </td> - </tr> - - <tr> - <td class="property" id="job-config-rewriter-regex">job.config.rewriter.<br><span class="rewriter">rewriter-name</span>.regex</td> - <td class="default"></td> - <td class="description"> - A regular expression specifying which topics you want to consume within the Kafka system - <a href="#job-config-rewriter-system" class="property">job.config.rewriter.*.system</a>. - Any topics matched by this regular expression will be consumed <em>in addition to</em> any - topics you specify with <a href="#task-inputs" class="property">task.inputs</a>. - </td> - </tr> - - <tr> - <td class="property" id="job-config-rewriter-config">job.config.rewriter.<br><span class="rewriter">rewriter-name</span>.config.*</td> - <td class="default"></td> - <td class="description"> - Any properties specified within this namespace are applied to the configuration of streams - that match the regex in - <a href="#job-config-rewriter-regex" class="property">job.config.rewriter.*.regex</a>. - For example, you can set <code>job.config.rewriter.*.config.samza.msg.serde</code> to configure - the deserializer for messages in the matching streams, which is equivalent to setting - <a href="#systems-samza-msg-serde" class="property">systems.*.streams.*.samza.msg.serde</a> - for each topic that matches the regex. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="state"><a href="../container/state-management.html">Storage and State Management</a></th> - </tr> - - <tr> - <td class="property" id="stores-factory">stores.<span class="store">store-name</span>.factory</td> - <td class="default"></td> - <td class="description"> - This property defines a store, Samza's mechanism for efficient - <a href="../container/state-management.html">stateful stream processing</a>. You can give a - store any <span class="store">store-name</span>, and use that name to get a reference to the - store in your stream task (call - <a href="../api/javadocs/org/apache/samza/task/TaskContext.html#getStore(java.lang.String)">TaskContext.getStore()</a> - in your task's - <a href="../api/javadocs/org/apache/samza/task/InitableTask.html#init(org.apache.samza.config.Config, org.apache.samza.task.TaskContext)">init()</a> - method). The value of this property is the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/storage/StorageEngineFactory.html">StorageEngineFactory</a>. - Samza currently ships with one storage engine implementation: - <dl> - <dt><code>org.apache.samza.storage.kv.KeyValueStorageEngineFactory</code></dt> - <dd>An on-disk storage engine with a key-value interface, implemented using - <a href="https://code.google.com/p/leveldb/";>LevelDB</a>. It supports fast random-access - reads and writes, as well as range queries on keys. LevelDB can be configured with - various <a href="#keyvalue">additional tuning parameters</a>.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="stores-key-serde">stores.<span class="store">store-name</span>.key.serde</td> - <td class="default"></td> - <td class="description"> - If the storage engine expects keys in the store to be simple byte arrays, this - <a href="../container/serialization.html">serde</a> allows the stream task to access the - store using another object type as key. The value of this property must be a - <span class="serde">serde-name</span> that is registered with - <a href="#serializers-registry-class" class="property">serializers.registry.*.class</a>. - If this property is not set, keys are passed unmodified to the storage engine - (and the <a href="#stores-changelog">changelog stream</a>, if appropriate). - </td> - </tr> - - <tr> - <td class="property" id="stores-msg-serde">stores.<span class="store">store-name</span>.msg.serde</td> - <td class="default"></td> - <td class="description"> - If the storage engine expects values in the store to be simple byte arrays, this - <a href="../container/serialization.html">serde</a> allows the stream task to access the - store using another object type as value. The value of this property must be a - <span class="serde">serde-name</span> that is registered with - <a href="#serializers-registry-class" class="property">serializers.registry.*.class</a>. - If this property is not set, values are passed unmodified to the storage engine - (and the <a href="#stores-changelog">changelog stream</a>, if appropriate). - </td> - </tr> - - <tr> - <td class="property" id="stores-changelog">stores.<span class="store">store-name</span>.changelog</td> - <td class="default"></td> - <td class="description"> - Samza stores are local to a container. If the container fails, the contents of the - store are lost. To prevent loss of data, you need to set this property to configure - a changelog stream: Samza then ensures that writes to the store are replicated to - this stream, and the store is restored from this stream after a failure. The value - of this property is given in the form - <span class="system">system-name</span>.<span class="stream">stream-name</span>. - Any output stream can be used as changelog, but you must ensure that only one job - ever writes to a given changelog stream (each instance of a job and each store - needs its own changelog stream). - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="keyvalue"> - Using LevelDB for key-value storage<br> - <span class="subtitle"> - (This section applies if you have set - <a href="#stores-factory" class="property">stores.*.factory</a> - <code>= org.apache.samza.storage.kv.KeyValueStorageEngineFactory</code>) - </span> - </th> - </tr> - - <tr> - <td class="property" id="stores-write-batch-size">stores.<span class="store">store-name</span>.<br>write.batch.size</td> - <td class="default">500</td> - <td class="description"> - For better write performance, the storage engine buffers writes and applies them - to the underlying store in a batch. If the same key is written multiple times - in quick succession, this buffer also deduplicates writes to the same key. This - property is set to the number of key/value pairs that should be kept in this - in-memory buffer, per task instance. The number cannot be greater than - <a href="#stores-object-cache-size" class="property">stores.*.object.cache.size</a>. - </td> - </tr> - - <tr> - <td class="property" id="stores-object-cache-size">stores.<span class="store">store-name</span>.<br>object.cache.size</td> - <td class="default">1000</td> - <td class="description"> - Samza maintains an additional cache in front of LevelDB for frequently-accessed - objects. This cache contains deserialized objects (avoiding the deserialization - overhead on cache hits), in contrast to the LevelDB block cache - (<a href="#stores-container-cache-size-bytes" class="property">stores.*.container.cache.size.bytes</a>), - which caches serialized objects. This property determines the number of objects - to keep in Samza's cache, per task instance. This same cache is also used for - write buffering (see <a href="#stores-write-batch-size" class="property">stores.*.write.batch.size</a>). - A value of 0 disables all caching and batching. - </td> - </tr> - - <tr> - <td class="property" id="stores-container-cache-size-bytes">stores.<span class="store">store-name</span>.container.<br>cache.size.bytes</td> - <td class="default">104857600</td> - <td class="description"> - The size of LevelDB's block cache in bytes, per container. If there are several - task instances within one container, each is given a proportional share of this cache. - Note that this is an off-heap memory allocation, so the container's total memory use - is the maximum JVM heap size <em>plus</em> the size of this cache. - </td> - </tr> - - <tr> - <td class="property" id="stores-container-write-buffer-size-bytes">stores.<span class="store">store-name</span>.container.<br>write.buffer.size.bytes</td> - <td class="default">33554432</td> - <td class="description"> - The amount of memory (in bytes) that LevelDB uses for buffering writes before they are - written to disk, per container. If there are several task instances within one - container, each is given a proportional share of this buffer. This setting also - determines the size of LevelDB's segment files. - </td> - </tr> - - <tr> - <td class="property" id="stores-compaction-delete-threshold">stores.<span class="store">store-name</span>.<br>compaction.delete.threshold</td> - <td class="default">-1</td> - <td class="description"> - Setting this property forces a LevelDB compaction to be performed after a certain - number of keys have been deleted from the store. This is used to work around - <a href="https://issues.apache.org/jira/browse/SAMZA-254";>performance issues</a> - in certain workloads. - </td> - </tr> - - <tr> - <td class="property" id="stores-leveldb-compression">stores.<span class="store">store-name</span>.<br>leveldb.compression</td> - <td class="default">snappy</td> - <td class="description"> - This property controls whether LevelDB should compress data on disk and in the - block cache. The following values are valid: - <dl> - <dt><code>snappy</code></dt> - <dd>Compress data using the <a href="https://code.google.com/p/snappy/";>Snappy</a> codec.</dd> - <dt><code>none</code></dt> - <dd>Do not compress data.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="stores-leveldb-block-size-bytes">stores.<span class="store">store-name</span>.<br>leveldb.block.size.bytes</td> - <td class="default">4096</td> - <td class="description"> - If compression is enabled, LevelDB groups approximately this many uncompressed bytes - into one compressed block. You probably don't need to change this property. - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="yarn"> - Running your job on a <a href="../jobs/yarn-jobs.html">YARN</a> cluster<br> - <span class="subtitle"> - (This section applies if you have set - <a href="#job-factory-class" class="property">job.factory.class</a> - <code>= org.apache.samza.job.yarn.YarnJobFactory</code>) - </span> - </th> - </tr> - - <tr> - <td class="property" id="yarn-package-path">yarn.package.path</td> - <td class="default"></td> - <td class="description"> - <strong>Required for YARN jobs:</strong> The URL from which the job package can - be downloaded, for example a <code>http://</code> or <code>hdfs://</code> URL. - The job package is a .tar.gz file with a - <a href="../jobs/packaging.html">specific directory structure</a>. - </td> - </tr> - - <tr> - <td class="property" id="yarn-container-count">yarn.container.count</td> - <td class="default">1</td> - <td class="description"> - The number of YARN containers to request for running your job. This is the main parameter - for controlling the scale (allocated computing resources) of your job: to increase the - parallelism of processing, you need to increase the number of containers. The minimum is one - container, and the maximum number of containers is the number of task instances (usually the - <a href="../container/samza-container.html#tasks-and-partitions">number of input stream partitions</a>). - Task instances are evenly distributed across the number of containers that you specify. - </td> - </tr> - - <tr> - <td class="property" id="yarn-container-memory-mb">yarn.container.memory.mb</td> - <td class="default">1024</td> - <td class="description"> - How much memory, in megabytes, to request from YARN per container of your job. Along with - <a href="#yarn-container-cpu-cores" class="property">yarn.container.cpu.cores</a>, this - property determines how many containers YARN will run on one machine. If the container - exceeds this limit, YARN will kill it, so it is important that the container's actual - memory use remains below the limit. The amount of memory used is normally the JVM heap - size (configured with <a href="#task-opts" class="property">task.opts</a>), plus the - size of any off-heap memory allocation (for example - <a href="#stores-container-cache-size-bytes" class="property">stores.*.container.cache.size.bytes</a>), - plus a safety margin to allow for JVM overheads. - </td> - </tr> - - <tr> - <td class="property" id="yarn-container-cpu-cores">yarn.container.cpu.cores</td> - <td class="default">1</td> - <td class="description"> - The number of CPU cores to request from YARN per container of your job. Each node in the - YARN cluster has a certain number of CPU cores available, so this number (along with - <a href="#yarn-container-memory-mb" class="property">yarn.container.memory.mb</a>) - determines how many containers can be run on one machine. Samza is - <a href="../container/event-loop.html">single-threaded</a> and designed to run on one - CPU core, so you shouldn't normally need to change this property. - </td> - </tr> - - <tr> - <td class="property" id="yarn-container-retry-count">yarn.container.<br>retry.count</td> - <td class="default">8</td> - <td class="description"> - If a container fails, it is automatically restarted by YARN. However, if a container keeps - failing shortly after startup, that indicates a deeper problem, so we should kill the job - rather than retrying indefinitely. This property determines the maximum number of times we are - willing to restart a failed container in quick succession (the time period is configured with - <a href="#yarn-container-retry-window-ms" class="property">yarn.container.retry.window.ms</a>). - Each container in the job is counted separately. If this property is set to 0, any failed - container immediately causes the whole job to fail. If it is set to a negative number, there - is no limit on the number of retries. - </td> - </tr> - - <tr> - <td class="property" id="yarn-container-retry-window-ms">yarn.container.<br>retry.window.ms</td> - <td class="default">300000</td> - <td class="description"> - This property determines how frequently a container is allowed to fail before we give up and - fail the job. If the same container has failed more than - <a href="#yarn-container-retry-count" class="property">yarn.container.retry.count</a> - times, and the time between failures was less than this property - <code>yarn.container.retry.window.ms</code> (in milliseconds), then we fail the job. - There is no limit to the number of times we will restart a container if the time between - failures is greater than <code>yarn.container.retry.window.ms</code>. - </td> - </tr> - - <tr> - <td class="property" id="yarn-am-container-memory-mb">yarn.am.container.<br>memory.mb</td> - <td class="default">1024</td> - <td class="description"> - Each Samza job has one special container, the - <a href="../yarn/application-master.html">ApplicationMaster</a> (AM), which manages the - execution of the job. This property determines how much memory, in megabytes, to request - from YARN for running the ApplicationMaster. - </td> - </tr> - - <tr> - <td class="property" id="yarn-am-opts">yarn.am.opts</td> - <td class="default"></td> - <td class="description"> - Any JVM options to include in the command line when executing the Samza - <a href="../yarn/application-master.html">ApplicationMaster</a>. For example, this can be - used to set the JVM heap size, to tune the garbage collector, or to enable remote debugging. - </td> - </tr> - - <tr> - <td class="property" id="yarn-am-poll-interval-ms">yarn.am.poll.interval.ms</td> - <td class="default">1000</td> - <td class="description"> - THe Samza ApplicationMaster sends regular heartbeats to the YARN ResourceManager - to confirm that it is alive. This property determines the time (in milliseconds) - between heartbeats. - </td> - </tr> - - <tr> - <td class="property" id="yarn-am-jmx-enabled">yarn.am.jmx.enabled</td> - <td class="default">true</td> - <td class="description"> - Determines whether a JMX server should be started on this job's YARN ApplicationMaster - (<code>true</code> or <code>false</code>). - </td> - </tr> - - <tr> - <th colspan="3" class="section" id="metrics"><a href="../container/metrics.html">Metrics</a></th> - </tr> - - <tr> - <td class="property" id="metrics-reporter-class">metrics.reporter.<br><span class="reporter">reporter-name</span>.class</td> - <td class="default"></td> - <td class="description"> - Samza automatically tracks various metrics which are useful for monitoring the health - of a job, and you can also track <a href="../container/metrics.html">your own metrics</a>. - With this property, you can define any number of <em>metrics reporters</em> which send - the metrics to a system of your choice (for graphing, alerting etc). You give each reporter - an arbitrary <span class="reporter">reporter-name</span>. To enable the reporter, you need - to reference the <span class="reporter">reporter-name</span> in - <a href="#metrics-reporters" class="property">metrics.reporters</a>. - The value of this property is the fully-qualified name of a Java class that implements - <a href="../api/javadocs/org/apache/samza/metrics/MetricsReporterFactory.html">MetricsReporterFactory</a>. - Samza ships with these implementations by default: - <dl> - <dt><code>org.apache.samza.metrics.reporter.JmxReporterFactory</code></dt> - <dd>With this reporter, every container exposes its own metrics as JMX MBeans. The JMX - server is started on a <a href="../container/jmx.html">random port</a> to avoid - collisions between containers running on the same machine.</dd> - <dt><code>org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory</code></dt> - <dd>This reporter sends the latest values of all metrics as messages to an output - stream once per minute. The output stream is configured with - <a href="#metrics-reporter-stream" class="property">metrics.reporter.*.stream</a> - and it can use any system supported by Samza.</dd> - </dl> - </td> - </tr> - - <tr> - <td class="property" id="metrics-reporters">metrics.reporters</td> - <td class="default"></td> - <td class="description"> - If you have defined any metrics reporters with - <a href="#metrics-reporter-class" class="property">metrics.reporter.*.class</a>, you - need to list them here in order to enable them. The value of this property is a - comma-separated list of <span class="reporter">reporter-name</span> tokens. - </td> - </tr> - - <tr> - <td class="property" id="metrics-reporter-stream">metrics.reporter.<br><span class="reporter">reporter-name</span>.stream</td> - <td class="default"></td> - <td class="description"> - If you have registered the metrics reporter - <a href="#metrics-reporter-class" class="property">metrics.reporter.*.class</a> - <code>= org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory</code>, - you need to set this property to configure the output stream to which the metrics data - should be sent. The stream is given in the form - <span class="system">system-name</span>.<span class="stream">stream-name</span>, - and the system must be defined in the job configuration. It's fine for many different jobs - to publish their metrics to the same metrics stream. Samza defines a simple - <a href="../container/metrics.html">JSON encoding</a> for metrics; in order to use this - encoding, you also need to configure a serde for the metrics stream: - <ul> - <li><a href="#systems-samza-msg-serde" class="property">systems.*.streams.*.samza.msg.serde</a> - <code>= metrics-serde</code> (replacing the asterisks with the - <span class="system">system-name</span> and <span class="stream">stream-name</span> - of the metrics stream)</li> - <li><a href="#serializers-registry-class" class="property">serializers.registry.metrics-serde.class</a> - <code>= org.apache.samza.serializers.MetricsSnapshotSerdeFactory</code> - (registering the serde under a <span class="serde">serde-name</span> of - <code>metrics-serde</code>)</li> - </ul> - </td> - </tr> - </tbody> - </table> - </body> -</html>
http://git-wip-us.apache.org/repos/asf/incubator-samza/blob/1e2cfe22/docs/learn/documentation/0.7.0/jobs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/0.7.0/jobs/configuration.md b/docs/learn/documentation/0.7.0/jobs/configuration.md deleted file mode 100644 index 8bcc433..0000000 --- a/docs/learn/documentation/0.7.0/jobs/configuration.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -layout: page -title: Configuration ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - -All Samza jobs have a configuration file that defines the job. A very basic configuration file looks like this: - -{% highlight jproperties %} -# Job -job.factory.class=samza.job.local.ThreadJobFactory -job.name=hello-world - -# Task -task.class=samza.task.example.MyJavaStreamerTask -task.inputs=example-system.example-stream - -# Serializers -serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory -serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory - -# Systems -systems.example-system.samza.factory=samza.stream.example.ExampleConsumerFactory -systems.example-system.samza.key.serde=string -systems.example-system.samza.msg.serde=json -{% endhighlight %} - -There are four major sections to a configuration file: - -1. The job section defines things like the name of the job, and whether to use the YarnJobFactory or ProcessJobFactory/ThreadJobFactory. -2. The task section is where you specify the class name for your [StreamTask](../api/overview.html). It's also where you define what the [input streams](../container/streams.html) are for your task. -3. The serializers section defines the classes of the [serdes](../container/serialization.html) used for serialization and deserialization of specific objects that are received and sent along different streams. -4. The system section defines systems that your StreamTask can read from along with the types of serdes used for sending keys and messages from that system. Usually, you'll define a Kafka system, if you're reading from Kafka, although you can also specify your own self-implemented Samza-compatible systems. See the [hello-samza example project](/startup/hello-samza/0.7.0)'s Wikipedia system for a good example of a self-implemented system. - -### Required Configuration - -Configuration keys that absolutely must be defined for a Samza job are: - -* `job.factory.class` -* `job.name` -* `task.class` -* `task.inputs` - -### Configuration Keys - -A complete list of configuration keys can be found on the [Configuration Table](configuration-table.html) page. - -## [Packaging »](packaging.html) http://git-wip-us.apache.org/repos/asf/incubator-samza/blob/1e2cfe22/docs/learn/documentation/0.7.0/jobs/job-runner.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/0.7.0/jobs/job-runner.md b/docs/learn/documentation/0.7.0/jobs/job-runner.md deleted file mode 100644 index c2f6b09..0000000 --- a/docs/learn/documentation/0.7.0/jobs/job-runner.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -layout: page -title: JobRunner ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - -Samza jobs are started using a script called run-job.sh. - -{% highlight bash %} -samza-example/target/bin/run-job.sh \ - --config-factory=samza.config.factories.PropertiesConfigFactory \ - --config-path=file://$PWD/config/hello-world.properties -{% endhighlight %} - -You provide two parameters to the run-job.sh script. One is the config location, and the other is a factory class that is used to read your configuration file. The run-job.sh script is actually executing a Samza class called JobRunner. The JobRunner uses your ConfigFactory to get a Config object from the config path. - -{% highlight java %} -public interface ConfigFactory { - Config getConfig(URI configUri); -} -{% endhighlight %} - -The Config object is just a wrapper around Map<String, String>, with some nice helper methods. Out of the box, Samza ships with the PropertiesConfigFactory, but developers can implement any kind of ConfigFactory they wish. - -Once the JobRunner gets your configuration, it gives your configuration to the StreamJobFactory class defined by the "job.factory" property. Samza ships with three job factory implementations: ThreadJobFactory, ProcessJobFactory and YarnJobFactory. The StreamJobFactory's responsibility is to give the JobRunner a job that it can run. - -{% highlight java %} -public interface StreamJob { - StreamJob submit(); - - StreamJob kill(); - - ApplicationStatus waitForFinish(long timeoutMs); - - ApplicationStatus waitForStatus(ApplicationStatus status, long timeoutMs); - - ApplicationStatus getStatus(); -} -{% endhighlight %} - -Once the JobRunner gets a job, it calls submit() on the job. This method is what tells the StreamJob implementation to start the SamzaContainer. In the case of LocalJobRunner, it uses a run-container.sh script to execute the SamzaContainer in a separate process, which will start one SamzaContainer locally on the machine that you ran run-job.sh on. - -This flow differs slightly when you use YARN, but we'll get to that later. - -## [Configuration »](configuration.html)
