Modified: samza/site/learn/documentation/latest/comparisons/storm.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/comparisons/storm.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/comparisons/storm.html (original) +++ samza/site/learn/documentation/latest/comparisons/storm.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/comparisons/storm.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/comparisons/storm";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -191,7 +189,7 @@ <p>When using a transactional spout with Trident (a requirement for achieving exactly-once semantics), parallelism is potentially reduced. Trident relies on a global ordering in its input streams — that is, ordering across all partitions of a stream, not just within one partion. This means that the topology’s input stream has to go through a single spout instance, effectively ignoring the partitioning of the input stream. This spout may become a bottleneck on high-volume streams. In Samza, all stream processing is parallel — there are no such choke points.</p> -<h3 id="deployment-&-execution">Deployment & Execution</h3> +<h3 id="deployment-execution">Deployment & Execution</h3> <p>A Storm cluster is composed of a set of nodes running a <em>Supervisor</em> daemon. The supervisor daemons talk to a single master node running a daemon called <em>Nimbus</em>. The Nimbus daemon is responsible for assigning work and managing resources in the cluster. See Storm’s <a href="https://github.com/nathanmarz/storm/wiki/Tutorial";>Tutorial</a> page for details. This is quite similar to YARN; though YARN is a bit more fully featured and intended to be multi-framework, Nimbus is better integrated with Storm.</p> @@ -221,7 +219,7 @@ <p>Samza is pretty immature, though it builds on solid components. YARN is fairly new, but is already being run on 3000+ node clusters at Yahoo!, and the project is under active development by both <a href="http://hortonworks.com/";>Hortonworks</a> and <a href="http://www.cloudera.com/content/cloudera/en/home.html";>Cloudera</a>. Kafka has a strong <a href="https://cwiki.apache.org/KAFKA/powered-by.html";>powered by</a> page, and has seen increased adoption recently. It’s also frequently used with Storm. Samza is a brand new project that is in use at LinkedIn. Our hope is that others will find it useful, and adopt it as well.</p> -<h3 id="buffering-&-latency">Buffering & Latency</h3> +<h3 id="buffering-latency">Buffering & Latency</h3> <p>Storm uses <a href="http://zeromq.org/";>ZeroMQ</a> for non-durable communication between bolts, which enables extremely low latency transmission of tuples. Samza does not have an equivalent mechanism, and always writes task output to a stream.</p> @@ -251,7 +249,7 @@ <p>Samza’s serialization and data model are both pluggable. We are not terribly opinionated about which approach is best.</p> -<h2 id="spark-streaming-»"><a href="spark-streaming.html">Spark Streaming »</a></h2> +<h2 id="spark-streaming"><a href="spark-streaming.html">Spark Streaming »</a></h2> </div> @@ -269,7 +267,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/comparisons/storm.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/comparisons/storm" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } });
Modified: samza/site/learn/documentation/latest/container/checkpointing.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/checkpointing.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/checkpointing.html (original) +++ samza/site/learn/documentation/latest/container/checkpointing.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/checkpointing.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/checkpointing";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -169,7 +167,7 @@ <p>For checkpoints to be effective, they need to be written somewhere where they will survive faults. Samza allows you to write checkpoints to the file system (using FileSystemCheckpointManager), but that doesn’t help if the machine fails and the container needs to be restarted on another machine. The most common configuration is to use Kafka for checkpointing. You can enable this with the following job configuration:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># The name of your job determines the name under which checkpoints will be stored</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># The name of your job determines the name under which checkpoints will be stored</span> <span class="na">job.name</span><span class="o">=</span><span class="s">example-job</span> <span class="c"># Define a system called "kafka" for consuming and producing to a Kafka cluster</span> @@ -180,17 +178,17 @@ <span class="na">task.checkpoint.system</span><span class="o">=</span><span class="s">kafka</span> <span class="c"># By default, a checkpoint is written every 60 seconds. You can change this if you like.</span> -<span class="na">task.commit.ms</span><span class="o">=</span><span class="s">60000</span></code></pre></div> +<span class="na">task.commit.ms</span><span class="o">=</span><span class="s">60000</span></code></pre></figure> <p>In this configuration, Samza writes checkpoints to a separate Kafka topic called __samza_checkpoint_<job-name>_<job-id> (in the example configuration above, the topic would be called __samza_checkpoint_example-job_1). Once per minute, Samza automatically sends a message to this topic, in which the current offsets of the input streams are encoded. When a Samza container starts up, it looks for the most recent offset message in this topic, and loads that checkpoint.</p> <p>Sometimes it can be useful to use checkpoints only for some input streams, but not for others. In this case, you can tell Samza to ignore any checkpointed offsets for a particular stream name:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Ignore any checkpoints for the topic "my-special-topic"</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Ignore any checkpoints for the topic "my-special-topic"</span> <span class="na">systems.kafka.streams.my-special-topic.samza.reset.offset</span><span class="o">=</span><span class="s">true</span> <span class="c"># Always start consuming "my-special-topic" at the oldest available offset</span> -<span class="na">systems.kafka.streams.my-special-topic.samza.offset.default</span><span class="o">=</span><span class="s">oldest</span></code></pre></div> +<span class="na">systems.kafka.streams.my-special-topic.samza.offset.default</span><span class="o">=</span><span class="s">oldest</span></code></pre></figure> <p>The following table explains the meaning of these configuration parameters:</p> @@ -232,14 +230,14 @@ <p>To inspect a job’s latest checkpoint, you need to specify your job’s config file, so that the tool knows which job it is dealing with:</p> -<div class="highlight"><pre><code class="bash">samza-example/target/bin/checkpoint-tool.sh <span class="se">\</span> - --config-path<span class="o">=</span>file:///path/to/job/config.properties</code></pre></div> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>samza-example/target/bin/checkpoint-tool.sh <span class="se">\</span> + --config-path<span class="o">=</span>file:///path/to/job/config.properties</code></pre></figure> <p>This command prints out the latest checkpoint in a properties file format. You can save the output to a file, and edit it as you wish. For example, to jump back to the oldest possible point in time, you can set all the offsets to 0. Then you can feed that properties file back into checkpoint-tool.sh and save the modified checkpoint:</p> -<div class="highlight"><pre><code class="bash">samza-example/target/bin/checkpoint-tool.sh <span class="se">\</span> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>samza-example/target/bin/checkpoint-tool.sh <span class="se">\</span> --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> - --new-offsets<span class="o">=</span>file:///path/to/new/offsets.properties</code></pre></div> + --new-offsets<span class="o">=</span>file:///path/to/new/offsets.properties</code></pre></figure> <p>Note that Samza only reads checkpoints on container startup. In order for your checkpoint change to take effect, you need to first stop the job, then save the modified offsets, and then start the job again. If you write a checkpoint while the job is running, it will most likely have no effect.</p> @@ -256,15 +254,15 @@ Here are few examples:</p> <p>In order to use the checkpoint callback a SystemConsumer needs to implement the CheckpointListener interface:</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">CheckpointListener</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">CheckpointListener</span> <span class="o">{</span> <span class="kt">void</span> <span class="nf">onCheckpoint</span><span class="o">(</span><span class="n">Map</span><span class="o"><</span><span class="n">SystemStreamPartition</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="n">offsets</span><span class="o">);</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>For the SystemConsumers which implement this interface Samza will invoke onCheckpoint() callback every time OffsetManager checkpoints. Checkpoints are done per task, and ‘offsets’ are all the offsets Samza checkpoints for a task, and these are the offsets which will be passed to the consumer on restart. Note that the callback will happen after the checkpoint and is <strong>not</strong> atomic.</p> -<h2 id="state-management-»"><a href="state-management.html">State Management »</a></h2> +<h2 id="state-management"><a href="state-management.html">State Management »</a></h2> </div> @@ -282,7 +280,7 @@ Note that the callback will happen after <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/checkpointing.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/checkpointing" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/coordinator-stream.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/coordinator-stream.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/coordinator-stream.html (original) +++ samza/site/learn/documentation/latest/container/coordinator-stream.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/coordinator-stream.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/coordinator-stream";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -160,14 +158,14 @@ <h3 id="coordinator-stream-naming">Coordinator Stream Naming</h3> <p>The naming convention is very similar to that of the checkpoint topic that get’s created.</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="s">"__samza_coordinator_%s_%s"</span> <span class="n">format</span> <span class="o">(</span><span class="n">jobName</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s">"_"</span><span class="o">,</span> <span class="s">"-"</span><span class="o">),</span> <span class="n">jobId</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s">"_"</span><span class="o">,</span> <span class="s">"-"</span><span class="o">))</span> +<div class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="s">"__samza_coordinator_%s_%s"</span> <span class="n">format</span> <span class="o">(</span><span class="n">jobName</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s">"_"</span><span class="o">,</span> <span class="s">"-"</span><span class="o">),</span> <span class="n">jobId</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s">"_"</span><span class="o">,</span> <span class="s">"-"</span><span class="o">))</span> </code></pre></div> <h3 id="coordinator-stream-message-model">Coordinator Stream Message Model</h3> <p>Coordinator stream messages are modeled as key/value pairs. The key is a list of well defined fields: <em>version</em>, <em>type</em>, and <em>key</em>. The value is a <em>map</em>. There are some pre-defined fields (such as timestamp, host, etc) for the value map, which are common to all messages.</p> <p>The full structure for a CoordinatorStreamMessage is:</p> -<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="err">key</span> <span class="err">=></span> <span class="p">[</span><span class="s2">"<version-number>"</span><span class="p">,</span> <span class="s2">"<message-type>"</span><span class="p">,</span> <span class="s2">"<key>"</span><span class="p">]</span> +<div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="err">key</span> <span class="err">=></span> <span class="p">[</span><span class="s2">"<version-number>"</span><span class="p">,</span> <span class="s2">"<message-type>"</span><span class="p">,</span> <span class="s2">"<key>"</span><span class="p">]</span> <span class="err">message</span> <span class="err">=></span> <span class="p">{</span> <span class="nt">"host"</span><span class="p">:</span> <span class="s2">"<hostname>"</span><span class="p">,</span> @@ -198,7 +196,7 @@ border-left: 0; border-right: 0; }</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"> table td.property, table td.default { +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span> table td.property, table td.default { white-space: nowrap; } @@ -242,11 +240,11 @@ <p>Samza provides a command line tool to write Job Configuration messages to the coordinator stream. The tool can be used as follows:</p> -<div class="highlight"><pre><code class="bash">samza-example/target/bin/run-coordinator-stream-writer.sh <span class="se">\</span> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>samza-example/target/bin/run-coordinator-stream-writer.sh <span class="se">\</span> --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> - --type <span class="nb">set</span>-config <span class="se">\</span> + --type set-config <span class="se">\</span> --key job.container.count <span class="se">\</span> - --value 8</code></pre></div> + --value <span class="m">8</span></code></pre></figure> <h2 id="job-coordinator"><a name="JobCoordinator"></a>Job Coordinator</h2> @@ -277,7 +275,7 @@ <p>For other interesting features that can leverage this model, please refer the <a href="https://issues.apache.org/jira/secure/attachment/12670650/DESIGN-SAMZA-348-1.pdf";>design document</a>.</p> -<h2 id="event-loop-»"><a href="event-loop.html">Event Loop »</a></h2> +<h2 id="event-loop"><a href="event-loop.html">Event Loop »</a></h2> </div> @@ -295,7 +293,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/coordinator-stream.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/coordinator-stream" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/event-loop.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/event-loop.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/event-loop.html (original) +++ samza/site/learn/documentation/latest/container/event-loop.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/event-loop.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/event-loop";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -172,7 +170,7 @@ <p>The container does this, in a loop, until it is shut down.</p> -<h3 id="semantics-for-synchronous-tasks-v.s.-asynchronous-tasks">Semantics for Synchronous Tasks v.s. Asynchronous Tasks</h3> +<h3 id="semantics-for-synchronous-tasks-v-s-asynchronous-tasks">Semantics for Synchronous Tasks v.s. Asynchronous Tasks</h3> <p>The semantics of the event loop differs when running synchronous tasks and asynchronous tasks:</p> @@ -199,7 +197,7 @@ <p>A concrete example is a set of StreamTasks that all want to share the same try/catch logic in their process() method. A StreamTask can be implemented that wraps the original StreamTasks, and surrounds the original process() call with the appropriate try/catch logic. For more details, see <a href="https://issues.apache.org/jira/browse/SAMZA-437";>this discussion</a>.</p> -<h2 id="metrics-»"><a href="metrics.html">Metrics »</a></h2> +<h2 id="metrics"><a href="metrics.html">Metrics »</a></h2> </div> @@ -217,7 +215,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/event-loop.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/event-loop" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/jmx.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/jmx.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/jmx.html (original) +++ samza/site/learn/documentation/latest/container/jmx.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/jmx.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/jmx";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -151,18 +149,18 @@ <p>You can tell Samza to publish its internal <a href="metrics.html">metrics</a>, and any custom metrics you define, as JMX MBeans. To enable this, set the following properties in your job configuration:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Define a Samza metrics reporter called "jmx", which publishes to JMX</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Define a Samza metrics reporter called "jmx", which publishes to JMX</span> <span class="na">metrics.reporter.jmx.class</span><span class="o">=</span><span class="s">org.apache.samza.metrics.reporter.JmxReporterFactory</span> <span class="c"># Use it (if you have multiple reporters defined, separate them with commas)</span> -<span class="na">metrics.reporters</span><span class="o">=</span><span class="s">jmx</span></code></pre></div> +<span class="na">metrics.reporters</span><span class="o">=</span><span class="s">jmx</span></code></pre></figure> <p>JMX needs to be configured to use a specific port, but in a distributed environment, there is no way of knowing in advance which ports are available on the machines running your containers. Therefore Samza chooses the JMX port randomly. If you need to connect to it, you can find the port by looking in the container’s logs, which report the JMX server details as follows:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text">2014-06-02 21:50:17 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are samza-grid-1234.example.com +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>2014-06-02 21:50:17 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are samza-grid-1234.example.com 2014-06-02 21:50:17 JmxServer [INFO] Started JmxServer registry port=50214 server port=50215 url=service:jmx:rmi://localhost:50215/jndi/rmi://localhost:50214/jmxrmi 2014-06-02 21:50:17 JmxServer [INFO] If you are tunneling, you might want to try JmxServer registry port=50214 server port=50215 url=service:jmx:rmi://samza-grid-1234.example.com:50215/jndi/rmi://samza-grid-1234.example.com:50214/jmxrmi </code></pre></div> -<h2 id="jobrunner-»"><a href="../jobs/job-runner.html">JobRunner »</a></h2> +<h2 id="jobrunner"><a href="../jobs/job-runner.html">JobRunner »</a></h2> </div> @@ -180,7 +178,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/jmx.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/jmx" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/metrics-table.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/metrics-table.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/metrics-table.html (original) +++ samza/site/learn/documentation/latest/container/metrics-table.html Fri May 25 22:28:38 2018 @@ -659,10 +659,18 @@ </tr> <tr> <td><span class="store">store-name</span>-messages-restored</td> + <td>DEPRECATED: use <code>restored-messages</code> instead. Number of restored messages (key-value pairs) from the change log.</td> + </tr> + <tr> + <td><span class="store">store-name</span>-restored-messages</td> <td>Number of restored messages (key-value pairs) from the change log</td> </tr> <tr> <td><span class="store">store-name</span>-message-bytes</td> + <td>DEPRECATED: use <code>restored-bytes</code> instead. Total amount of bytes (key and value size) restored from the change log</td> + </tr> + <tr> + <td><span class="store">store-name</span>-restored-bytes</td> <td>Total amount of bytes (key and value size) restored from the change log</td> </tr> Modified: samza/site/learn/documentation/latest/container/metrics.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/metrics.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/metrics.html (original) +++ samza/site/learn/documentation/latest/container/metrics.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/metrics.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/metrics";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -153,7 +151,7 @@ <p>To set up your job to publish metrics to Kafka, you can use the following configuration:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Define a metrics reporter called "snapshot", which publishes metrics</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Define a metrics reporter called "snapshot", which publishes metrics</span> <span class="c"># every 60 seconds.</span> <span class="na">metrics.reporters</span><span class="o">=</span><span class="s">snapshot</span> <span class="na">metrics.reporter.snapshot.class</span><span class="o">=</span><span class="s">org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory</span> @@ -164,11 +162,11 @@ <span class="c"># Encode metrics data as JSON.</span> <span class="na">serializers.registry.metrics.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.MetricsSnapshotSerdeFactory</span> -<span class="na">systems.kafka.streams.metrics.samza.msg.serde</span><span class="o">=</span><span class="s">metrics</span></code></pre></div> +<span class="na">systems.kafka.streams.metrics.samza.msg.serde</span><span class="o">=</span><span class="s">metrics</span></code></pre></figure> <p>With this configuration, the job automatically sends several JSON-encoded messages to the “metrics” topic in Kafka every 60 seconds. The messages look something like this:</p> -<div class="highlight"><pre><code class="json"><span class="p">{</span> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> <span class="nt">"header"</span><span class="p">:</span> <span class="p">{</span> <span class="nt">"container-name"</span><span class="p">:</span> <span class="s2">"samza-container-0"</span><span class="p">,</span> <span class="nt">"host"</span><span class="p">:</span> <span class="s2">"samza-grid-1234.example.com"</span><span class="p">,</span> @@ -193,7 +191,7 @@ <span class="nt">"window-skipped"</span><span class="p">:</span> <span class="mi">77955</span> <span class="p">}</span> <span class="p">}</span> -<span class="p">}</span></code></pre></div> +<span class="p">}</span></code></pre></figure> <p>There is a separate message for each task instance, and the header tells you the job name, job ID and partition of the task. The metrics allow you to see how many messages have been processed and sent, the current offset in the input stream partition, and other details. There are additional messages which give you metrics about the JVM (heap size, garbage collection information, threads etc.), internal metrics of the Kafka producers and consumers, and more. The list of all metrics emitted by samza is shown <a href="metrics-table.html">here</a>.</p> @@ -201,7 +199,7 @@ <p>You can register your custom metrics through a <a href="../api/javadocs/org/apache/samza/metrics/MetricsRegistry.html">MetricsRegistry</a>. Your stream task needs to implement <a href="../api/javadocs/org/apache/samza/task/InitableTask.html">InitableTask</a>, so that you can get the metrics registry from the <a href="../api/javadocs/org/apache/samza/task/TaskContext.html">TaskContext</a>. This simple example shows how to count the number of messages processed by your task:</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyJavaStreamTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">InitableTask</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyJavaStreamTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">InitableTask</span> <span class="o">{</span> <span class="kd">private</span> <span class="n">Counter</span> <span class="n">messageCount</span><span class="o">;</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> @@ -215,13 +213,13 @@ <span class="n">TaskCoordinator</span> <span class="n">coordinator</span><span class="o">)</span> <span class="o">{</span> <span class="n">messageCount</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> <span class="o">}</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>Samza currently supports three kinds of metrics: <a href="../api/javadocs/org/apache/samza/metrics/Counter.html">counters</a>, <a href="../api/javadocs/org/apache/samza/metrics/Gauge.html">gauges</a> and <a href="../api/javadocs/org/apache/samza/metrics/Timer.html">timer</a>. Use a counter when you want to track how often something occurs, a gauge when you want to report the level of something, such as the size of a buffer, and a timer when you want to know how much time the block of code spends. Each task instance (for each input stream partition) gets its own set of metrics.</p> <p>If you want to report metrics in some other way, e.g. directly to a graphing system (without going via Kafka), you can implement a <a href="../api/javadocs/org/apache/samza/metrics/MetricsReporterFactory.html">MetricsReporterFactory</a> and reference it in your job configuration.</p> -<h2 id="jmx-»"><a href="jmx.html">JMX »</a></h2> +<h2 id="jmx"><a href="jmx.html">JMX »</a></h2> </div> @@ -239,7 +237,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/metrics.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/metrics" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/samza-container.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/samza-container.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/samza-container.html (original) +++ samza/site/learn/documentation/latest/container/samza-container.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/samza-container.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/samza-container";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -168,10 +166,10 @@ <p>When the container starts, it creates instances of the <a href="../api/overview.html">task class</a> that you’ve written. If the task class implements the <a href="../api/javadocs/org/apache/samza/task/InitableTask.html">InitableTask</a> interface, the SamzaContainer will also call the init() method.</p> -<div class="highlight"><pre><code class="java"><span class="cm">/** Implement this if you want a callback when your task starts up. */</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="cm">/** Implement this if you want a callback when your task starts up. */</span> <span class="kd">public</span> <span class="kd">interface</span> <span class="nc">InitableTask</span> <span class="o">{</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">);</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>By default, how many instances of your task class are created depends on the number of partitions in the job’s input streams. If your Samza job has ten partitions, there will be ten instantiations of your task class: one for each partition. The first task instance will receive all messages for partition one, the second instance will receive all messages for partition two, and so on.</p> @@ -234,11 +232,11 @@ <p>After 0.10.0, Samza supports broadcast streams. You can assign partitions from some streams to all the tasks, by appending the hash tag, and the partition number or the partition number range. For example, you want all the tasks can consume partition 0 and 1 from a stream called broadcast-stream-1, and partition 2 from a stream called broadcast-stream-2. You now can configure:</p> -<div class="highlight"><pre><code class="jproperties"><span class="na">task.broadcast.inputs</span><span class="o">=</span><span class="s">yourSystem.broadcast-stream-1#[0-1], yourSystem.broadcast-stream-2#2</span></code></pre></div> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">task.broadcast.inputs</span><span class="o">=</span><span class="s">yourSystem.broadcast-stream-1#[0-1], yourSystem.broadcast-stream-2#2</span></code></pre></figure> <p>If you use “[]”, you are specifying a range for partitions.</p> -<h2 id="streams-»"><a href="streams.html">Streams »</a></h2> +<h2 id="streams"><a href="streams.html">Streams »</a></h2> </div> @@ -256,7 +254,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/samza-container.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/samza-container" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/serialization.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/serialization.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/serialization.html (original) +++ samza/site/learn/documentation/latest/container/serialization.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/serialization.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/serialization";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -157,7 +155,7 @@ <p>You can use whatever makes sense for your job; Samza doesn’t impose any particular data model or serialization scheme on you. However, the cleanest solution is usually to use Samza’s serde layer. The following configuration example shows how to use it.</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Define a system called "kafka"</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Define a system called "kafka"</span> <span class="na">systems.kafka.samza.factory</span><span class="o">=</span><span class="s">org.apache.samza.system.kafka.KafkaSystemFactory</span> <span class="c"># The job is going to consume a topic called "PageViewEvent" from the "kafka" system</span> @@ -179,7 +177,7 @@ <span class="na">stores.LastPageViewPerUser.factory</span><span class="o">=</span><span class="s">org.apache.samza.storage.kv.KeyValueStorageEngineFactory</span> <span class="na">stores.LastPageViewPerUser.changelog</span><span class="o">=</span><span class="s">kafka.last-page-view-per-user</span> <span class="na">stores.LastPageViewPerUser.key.serde</span><span class="o">=</span><span class="s">integer</span> -<span class="na">stores.LastPageViewPerUser.msg.serde</span><span class="o">=</span><span class="s">json</span></code></pre></div> +<span class="na">stores.LastPageViewPerUser.msg.serde</span><span class="o">=</span><span class="s">json</span></code></pre></figure> <p>Each serde is defined with a factory class. Samza comes with several builtin serdes for UTF-8 strings, binary-encoded integers, JSON and more. The following is a comprehensive list of supported serdes in Samza. <style> @@ -192,7 +190,7 @@ border-left: 0; border-right: 0; }</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"> table td.property, table td.default { +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span> table td.property, table td.default { white-space: nowrap; } @@ -244,7 +242,7 @@ <p>All the Samza APIs for sending and receiving messages are typed as <em>Object</em>. This means that you have to cast messages to the correct type before you can use them. It’s a little bit more code, but it has the advantage that Samza is not restricted to any particular data model.</p> -<h2 id="checkpointing-»"><a href="checkpointing.html">Checkpointing »</a></h2> +<h2 id="checkpointing"><a href="checkpointing.html">Checkpointing »</a></h2> </div> @@ -262,7 +260,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/serialization.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/serialization" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/state-management.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/state-management.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/state-management.html (original) +++ samza/site/learn/documentation/latest/container/state-management.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/state-management.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/state-management";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -270,7 +268,7 @@ <p>To use a key-value store in your job, add the following to your job config:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Use the key-value store implementation for a store called "my-store"</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Use the key-value store implementation for a store called "my-store"</span> <span class="na">stores.my-store.factory</span><span class="o">=</span><span class="s">org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory</span> <span class="c"># Use the Kafka topic "my-store-changelog" as the changelog stream for this store.</span> @@ -281,13 +279,13 @@ <span class="c"># Encode keys and values in the store as UTF-8 strings.</span> <span class="na">serializers.registry.string.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.StringSerdeFactory</span> <span class="na">stores.my-store.key.serde</span><span class="o">=</span><span class="s">string</span> -<span class="na">stores.my-store.msg.serde</span><span class="o">=</span><span class="s">string</span></code></pre></div> +<span class="na">stores.my-store.msg.serde</span><span class="o">=</span><span class="s">string</span></code></pre></figure> <p>See the <a href="serialization.html">serialization section</a> for more information on the <em>serde</em> options.</p> <p>Here is a simple example that writes every incoming message to the store:</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatefulTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">InitableTask</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatefulTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">InitableTask</span> <span class="o">{</span> <span class="kd">private</span> <span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="n">store</span><span class="o">;</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> @@ -299,18 +297,18 @@ <span class="n">TaskCoordinator</span> <span class="n">coordinator</span><span class="o">)</span> <span class="o">{</span> <span class="n">store</span><span class="o">.</span><span class="na">put</span><span class="o">((</span><span class="n">String</span><span class="o">)</span> <span class="n">envelope</span><span class="o">.</span><span class="na">getKey</span><span class="o">(),</span> <span class="o">(</span><span class="n">String</span><span class="o">)</span> <span class="n">envelope</span><span class="o">.</span><span class="na">getMessage</span><span class="o">());</span> <span class="o">}</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>Here is the complete key-value store API:</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KeyValueStore</span><span class="o"><</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">></span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KeyValueStore</span><span class="o"><</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">></span> <span class="o">{</span> <span class="n">V</span> <span class="nf">get</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">);</span> <span class="kt">void</span> <span class="nf">put</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">,</span> <span class="n">V</span> <span class="n">value</span><span class="o">);</span> <span class="kt">void</span> <span class="nf">putAll</span><span class="o">(</span><span class="n">List</span><span class="o"><</span><span class="n">Entry</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">>></span> <span class="n">entries</span><span class="o">);</span> <span class="kt">void</span> <span class="nf">delete</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">);</span> - <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="n">range</span><span class="o">(</span><span class="n">K</span> <span class="n">from</span><span class="o">,</span> <span class="n">K</span> <span class="n">to</span><span class="o">);</span> - <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="n">all</span><span class="o">();</span> -<span class="o">}</span></code></pre></div> + <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="nf">range</span><span class="o">(</span><span class="n">K</span> <span class="n">from</span><span class="o">,</span> <span class="n">K</span> <span class="n">to</span><span class="o">);</span> + <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="nf">all</span><span class="o">();</span> +<span class="o">}</span></code></pre></figure> <p>Additional configuration properties for the key-value store are documented in the <a href="../jobs/configuration-table.html#keyvalue-rocksdb">configuration reference</a>.</p> @@ -320,19 +318,19 @@ <p>Currently Samza provides a state storage tool which can recover the state store from the changelog stream to user-specified directory for reusing and debugging.</p> -<div class="highlight"><pre><code class="bash">samza-example/target/bin/state-storage-tool.sh <span class="se">\</span> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>samza-example/target/bin/state-storage-tool.sh <span class="se">\</span> --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> - --path<span class="o">=</span>directory/to/put/state/stores</code></pre></div> + --path<span class="o">=</span>directory/to/put/state/stores</code></pre></figure> <h4 id="read-the-value-from-a-running-rocksdb">Read the value from a running RocksDB</h4> <p>Samza also provides a tool to read the value from a running job’s RocksDB.</p> -<div class="highlight"><pre><code class="bash">samza-example/target/bin/read-rocksdb-tool.sh <span class="se">\</span> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>samza-example/target/bin/read-rocksdb-tool.sh <span class="se">\</span> --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> --db-path<span class="o">=</span>/tmp/nm-local-dir/state/test-state/Partition_0 <span class="se">\</span> - --db-name<span class="o">=</span><span class="nb">test</span>-state <span class="se">\</span> - --string-key<span class="o">=</span>a,b,c</code></pre></div> + --db-name<span class="o">=</span>test-state <span class="se">\</span> + --string-key<span class="o">=</span>a,b,c</code></pre></figure> <ul> <li><code>--config-path</code>(required): your job’s configuration file</li> @@ -348,8 +346,8 @@ <li>This only works with three kinds of keys: string, integer and long. This is because we can only accept those kinds of keys from the command line (it is really tricky to accept bytes, avro, json, etc from the command line). But it is also easy to use this tool programmatically (The key and value both are deserialized.)</li> </ul> -<div class="highlight"><pre><code class="bash">RocksDbKeyValueReader <span class="nv">kvReader</span> <span class="o">=</span> new RocksDbKeyValueReader<span class="o">(</span>dbName, pathOfdb, config<span class="o">)</span> -Object <span class="nv">value</span> <span class="o">=</span> kvReader.get<span class="o">(</span>key<span class="o">)</span></code></pre></div> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>RocksDbKeyValueReader <span class="nv">kvReader</span> <span class="o">=</span> new RocksDbKeyValueReader<span class="o">(</span>dbName, pathOfdb, config<span class="o">)</span> +Object <span class="nv">value</span> <span class="o">=</span> kvReader.get<span class="o">(</span>key<span class="o">)</span></code></pre></figure> <ul> <li>Because Samza job has some caches and buffers, you may not be able to see expected values (or even not be able to see any value, if all the data is buffered). Some of the related configuration are <code>stores.store-name.container.write.buffer.size.bytes</code>, <code>stores.store-name.write.batch.size</code>, <code>stores.store-name.object.cache.size</code>. You may want to set them to very small for testing.</li> @@ -419,7 +417,7 @@ Object <span class="nv">value</span> <sp <p>However, for non-idempotent operations such as counting, at-least-once delivery guarantees can give incorrect results. If a Samza task fails and is restarted, it may double-count some messages that were processed shortly before the failure. We are planning to address this limitation in a future release of Samza.</p> -<h2 id="windowing-»"><a href="windowing.html">Windowing »</a></h2> +<h2 id="windowing"><a href="windowing.html">Windowing »</a></h2> </div> @@ -437,7 +435,7 @@ Object <span class="nv">value</span> <sp <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/state-management.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/state-management" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/streams.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/streams.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/streams.html (original) +++ samza/site/learn/documentation/latest/container/streams.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/streams.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/streams";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -149,7 +147,7 @@ <p>The <a href="samza-container.html">samza container</a> reads and writes messages using the <a href="../api/javadocs/org/apache/samza/system/SystemConsumer.html">SystemConsumer</a> and <a href="../api/javadocs/org/apache/samza/system/SystemProducer.html">SystemProducer</a> interfaces. You can integrate any message broker with Samza by implementing these two interfaces.</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">SystemConsumer</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">SystemConsumer</span> <span class="o">{</span> <span class="kt">void</span> <span class="nf">start</span><span class="o">();</span> <span class="kt">void</span> <span class="nf">stop</span><span class="o">();</span> @@ -189,7 +187,7 @@ <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getKey</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getMessage</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>Out of the box, Samza supports Kafka (KafkaSystemConsumer and KafkaSystemProducer). However, any message bus system can be plugged in, as long as it can provide the semantics required by Samza, as described in the <a href="../api/javadocs/org/apache/samza/system/SystemConsumer.html">javadoc</a>.</p> @@ -209,7 +207,7 @@ <p>To plug in your own message chooser, you need to implement the <a href="../api/javadocs/org/apache/samza/system/chooser/MessageChooserFactory.html">MessageChooserFactory</a> interface, and set the “task.chooser.class” configuration to the fully-qualified class name of your implementation:</p> -<div class="highlight"><pre><code class="jproperties"><span class="na">task.chooser.class</span><span class="o">=</span><span class="s">com.example.samza.YourMessageChooserFactory</span></code></pre></div> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">task.chooser.class</span><span class="o">=</span><span class="s">com.example.samza.YourMessageChooserFactory</span></code></pre></figure> <h4 id="prioritizing-input-streams">Prioritizing input streams</h4> @@ -217,8 +215,8 @@ <p>Samza provides a mechanism to prioritize one stream over another by setting this configuration parameter: systems.<system>.streams.<stream>.samza.priority=<number>. For example:</p> -<div class="highlight"><pre><code class="jproperties"><span class="na">systems.kafka.streams.my-real-time-stream.samza.priority</span><span class="o">=</span><span class="s">2</span> -<span class="na">systems.kafka.streams.my-batch-stream.samza.priority</span><span class="o">=</span><span class="s">1</span></code></pre></div> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">systems.kafka.streams.my-real-time-stream.samza.priority</span><span class="o">=</span><span class="s">2</span> +<span class="na">systems.kafka.streams.my-batch-stream.samza.priority</span><span class="o">=</span><span class="s">1</span></code></pre></figure> <p>This declares that my-real-time-stream’s messages should be processed with higher priority than my-batch-stream’s messages. If my-real-time-stream has any messages available, they are processed first. Only if there are no messages currently waiting on my-real-time-stream, the Samza job continues processing my-batch-stream.</p> @@ -236,9 +234,9 @@ <p>To configure a stream called “my-bootstrap-stream” to be a fully-consumed bootstrap stream, use the following settings:</p> -<div class="highlight"><pre><code class="jproperties"><span class="na">systems.kafka.streams.my-bootstrap-stream.samza.bootstrap</span><span class="o">=</span><span class="s">true</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">systems.kafka.streams.my-bootstrap-stream.samza.bootstrap</span><span class="o">=</span><span class="s">true</span> <span class="na">systems.kafka.streams.my-bootstrap-stream.samza.reset.offset</span><span class="o">=</span><span class="s">true</span> -<span class="na">systems.kafka.streams.my-bootstrap-stream.samza.offset.default</span><span class="o">=</span><span class="s">oldest</span></code></pre></div> +<span class="na">systems.kafka.streams.my-bootstrap-stream.samza.offset.default</span><span class="o">=</span><span class="s">oldest</span></code></pre></figure> <p>The bootstrap=true parameter enables the bootstrap behavior (prioritization over other streams). The combination of reset.offset=true and offset.default=oldest tells Samza to always start reading the stream from the oldest offset, every time a container starts up (rather than starting to read from the most recent checkpoint).</p> @@ -250,11 +248,11 @@ <p>For example, if you want to read 100 messages in a row from each stream partition (regardless of the MessageChooser), you can use this configuration parameter:</p> -<div class="highlight"><pre><code class="jproperties"><span class="na">task.consumer.batch.size</span><span class="o">=</span><span class="s">100</span></code></pre></div> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">task.consumer.batch.size</span><span class="o">=</span><span class="s">100</span></code></pre></figure> <p>With this setting, Samza tries to read a message from the most recently used <a href="../api/javadocs/org/apache/samza/system/SystemStreamPartition.html">SystemStreamPartition</a>. This behavior continues either until no more messages are available for that SystemStreamPartition, or until the batch size has been reached. When that happens, Samza defers to the MessageChooser to determine the next message to process. It then again tries to continue consume from the chosen message’s SystemStreamPartition until the batch size is reached.</p> -<h2 id="serialization-»"><a href="serialization.html">Serialization »</a></h2> +<h2 id="serialization"><a href="serialization.html">Serialization »</a></h2> </div> @@ -272,7 +270,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/streams.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/streams" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/windowing.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/windowing.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/windowing.html (original) +++ samza/site/learn/documentation/latest/container/windowing.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/container/windowing.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/container/windowing";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -151,17 +149,17 @@ <p>Samza’s <em>windowing</em> feature provides a way for tasks to do something in regular time intervals, for example once per minute. To enable windowing, you just need to set one property in your job configuration:</p> -<div class="highlight"><pre><code class="jproperties"><span class="c"># Call the window() method every 60 seconds</span> -<span class="na">task.window.ms</span><span class="o">=</span><span class="s">60000</span></code></pre></div> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="c"># Call the window() method every 60 seconds</span> +<span class="na">task.window.ms</span><span class="o">=</span><span class="s">60000</span></code></pre></figure> <p>Next, your stream task needs to implement the <a href="../api/javadocs/org/apache/samza/task/WindowableTask.html">WindowableTask</a> interface. This interface defines a window() method which is called by Samza in the regular interval that you configured.</p> <p>For example, this is how you would implement a basic per-minute event counter:</p> -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">EventCounterTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">WindowableTask</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">EventCounterTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">WindowableTask</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">SystemStream</span> <span class="n">OUTPUT_STREAM</span> <span class="o">=</span> - <span class="k">new</span> <span class="nf">SystemStream</span><span class="o">(</span><span class="s">"kafka"</span><span class="o">,</span> <span class="s">"events-per-minute"</span><span class="o">);</span> + <span class="k">new</span> <span class="n">SystemStream</span><span class="o">(</span><span class="s">"kafka"</span><span class="o">,</span> <span class="s">"events-per-minute"</span><span class="o">);</span> <span class="kd">private</span> <span class="kt">int</span> <span class="n">eventsSeen</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> @@ -176,13 +174,13 @@ <span class="n">collector</span><span class="o">.</span><span class="na">send</span><span class="o">(</span><span class="k">new</span> <span class="n">OutgoingMessageEnvelope</span><span class="o">(</span><span class="n">OUTPUT_STREAM</span><span class="o">,</span> <span class="n">eventsSeen</span><span class="o">));</span> <span class="n">eventsSeen</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="o">}</span> -<span class="o">}</span></code></pre></div> +<span class="o">}</span></code></pre></figure> <p>If you need to send messages to output streams, you can use the <a href="../api/javadocs/org/apache/samza/task/MessageCollector.html">MessageCollector</a> object passed to the window() method. Please only use that MessageCollector object for sending messages, and don’t use it outside of the call to window().</p> <p>Note that Samza uses <a href="event-loop.html">single-threaded execution</a>, so the window() call can never happen concurrently with a process() call. This has the advantage that you don’t need to worry about thread safety in your code (no need to synchronize anything), but the downside that the window() call may be delayed if your process() method takes a long time to return.</p> -<h2 id="coordinator-stream-»"><a href="coordinator-stream.html">Coordinator Stream »</a></h2> +<h2 id="coordinator-stream"><a href="coordinator-stream.html">Coordinator Stream »</a></h2> </div> @@ -200,7 +198,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/container/windowing.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/container/windowing" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/hadoop/consumer.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/hadoop/consumer.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/hadoop/consumer.html (original) +++ samza/site/learn/documentation/latest/hadoop/consumer.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/consumer.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/consumer";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -178,7 +176,7 @@ <h3 id="basic-configuration">Basic Configuration</h3> <p>Here is a few of the basic configs to set up HdfsSystemConsumer:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"># The HDFS system consumer is implemented under the org.apache.samza.system.hdfs package, +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span># The HDFS system consumer is implemented under the org.apache.samza.system.hdfs package, # so use HdfsSystemFactory as the system factory for your system systems.hdfs-clickstream.samza.factory=org.apache.samza.system.hdfs.HdfsSystemFactory @@ -196,7 +194,7 @@ systems.hdfs-clickstream.partitioner.def <h3 id="security-configuration">Security Configuration</h3> <p>The following additional configs are required when accessing HDFS clusters that have kerberos enabled:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"># Use the SamzaYarnSecurityManagerFactory, which fetches and renews the Kerberos delegation tokens when the job is running in a secure environment. +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span># Use the SamzaYarnSecurityManagerFactory, which fetches and renews the Kerberos delegation tokens when the job is running in a secure environment. job.security.manager.factory=org.apache.samza.job.yarn.SamzaYarnSecurityManagerFactory # Kerberos principal @@ -208,11 +206,11 @@ yarn.kerberos.keytab=/tmp/keytab <h3 id="advanced-configuration">Advanced Configuration</h3> <p>Some of the advanced configuration you might need to set up:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"># Specify the group pattern for advanced partitioning. +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span># Specify the group pattern for advanced partitioning. systems.hdfs-clickstream.partitioner.defaultPartitioner.groupPattern=part-[id]-.* </code></pre></div> <p>The advanced partitioning goes beyond the basic assumption that each file is a partition. With advanced partitioning you can group files into partitions arbitrarily. For example, if you have a set of files as [part-01-a.avro, part-01-b.avro, part-02-a.avro, part-02-b.avro, part-03-a.avro] that you want to organize into three partitions as (part-01-a.avro, part-01-b.avro), (part-02-a.avro, part-02-b.avro), (part-03-a.avro), where the numbers in the middle act as a “group identifier”, you can then set this property to be “part-[id]-.<em>” (note that *</em>[id]** is a reserved term here, i.e. you have to literally put it as <strong>[id]</strong>). The partitioner will apply this pattern to all file names and extract the “group identifier” (“[id]” in the pattern), then use the “group identifier” to group files into partitions.</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"># Specify the type of files your job want to process (support avro only for now) +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span># Specify the type of files your job want to process (support avro only for now) systems.hdfs-clickstream.consumer.reader=avro # Max number of retries (per-partition) before the container fails. @@ -224,7 +222,7 @@ system.hdfs-clickstream.consumer.numMaxR <p><a href="https://issues.apache.org/jira/secure/attachment/12827670/HDFSSystemConsumer.pdf";>HdfsSystemConsumer design doc</a></p> -<h2 id="writing-to-hdfs-»"><a href="./producer.html">Writing to HDFS »</a></h2> +<h2 id="writing-to-hdfs"><a href="./producer.html">Writing to HDFS »</a></h2> </div> @@ -242,7 +240,7 @@ system.hdfs-clickstream.consumer.numMaxR <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/consumer.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/consumer" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/hadoop/overview.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/hadoop/overview.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/hadoop/overview.html (original) +++ samza/site/learn/documentation/latest/hadoop/overview.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/overview.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/overview";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -171,7 +169,7 @@ <p>During a job restart, batch processing behaves completely different from streaming. In batch, it is expected to be a re-run and all the internal streams, including intermediate, checkpoint and changelog streams, need to be fresh. Since some systems only support retention-based stream cleanup, e.g. Kafka without deletion enabled, Samza creates a new set of internal streams for each job run. To achieve this, Samza internally generates a unique <strong>run.id</strong> to each job run. The <strong>run.id</strong> is appended to the physical names of the internal streams, which will be used in the job in each run. Samza also performs due diligence to delete/purge the streams from previous run. The cleanup happens when the job is restarted.</p> -<h2 id="reading-from-hdfs-»"><a href="./consumer.html">Reading from HDFS »</a></h2> +<h2 id="reading-from-hdfs"><a href="./consumer.html">Reading from HDFS »</a></h2> </div> @@ -189,7 +187,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/overview.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/overview" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/hadoop/producer.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/hadoop/producer.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/hadoop/producer.html (original) +++ samza/site/learn/documentation/latest/hadoop/producer.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/producer.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/hadoop/producer";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -154,7 +152,7 @@ The last one writes out Avro data files <p>You can configure an HdfsSystemProducer like any other Samza system: using configuration keys and values set in a <code>job.properties</code> file. You might configure the system producer for use by your <code>StreamTasks</code> like this:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"># set the SystemFactory implementation to instantiate HdfsSystemProducer aliased to 'hdfs-clickstream' +<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span># set the SystemFactory implementation to instantiate HdfsSystemProducer aliased to 'hdfs-clickstream' systems.hdfs-clickstream.samza.factory=org.apache.samza.system.hdfs.HdfsSystemFactory # define a serializer/deserializer for the hdfs-clickstream system @@ -208,7 +206,7 @@ systems.hdfs-clickstream.producer.hdfs.w <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/producer.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/hadoop/producer" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/index.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/index.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/index.html (original) +++ samza/site/learn/documentation/latest/index.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/index.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -267,7 +265,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/index.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/architecture.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/architecture.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/architecture.html (original) +++ samza/site/learn/documentation/latest/introduction/architecture.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/introduction/architecture.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/introduction/architecture";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -227,7 +225,7 @@ <p>Let’s take a look at a real example: suppose we want to count the number of page views. In SQL, you would write something like:</p> -<div class="highlight"><pre><code class="sql"><span class="k">SELECT</span> <span class="n">user_id</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">PageViewEvent</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">user_id</span></code></pre></div> +<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span></span><span class="k">SELECT</span> <span class="n">user_id</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">PageViewEvent</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">user_id</span></code></pre></figure> <p>Although Samza doesn’t support SQL right now, the idea is the same. Two jobs are required to calculate this query: one to group messages by user ID, and the other to do the counting.</p> @@ -241,7 +239,7 @@ <p>By partitioning topics, and by breaking a stream process down into jobs and parallel tasks that run on multiple machines, Samza scales to streams with very high message throughput. By using YARN and Kafka, Samza achieves fault-tolerance: if a process or machine fails, it is automatically restarted on another machine and continues processing messages from the point where it left off.</p> -<h2 id="comparison-introduction-»"><a href="../comparisons/introduction.html">Comparison Introduction »</a></h2> +<h2 id="comparison-introduction"><a href="../comparisons/introduction.html">Comparison Introduction »</a></h2> </div> @@ -259,7 +257,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/architecture.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/architecture" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/background.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/background.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/background.html (original) +++ samza/site/learn/documentation/latest/introduction/background.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/introduction/background.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/introduction/background";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -149,7 +147,7 @@ <p>This page provides some background about stream processing, describes what Samza is, and why it was built.</p> -<h3 id="what-is-messaging?">What is messaging?</h3> +<h3 id="what-is-messaging">What is messaging?</h3> <p>Messaging systems are a popular way of implementing near-realtime asynchronous computation. Messages can be added to a message queue (ActiveMQ, RabbitMQ), pub-sub system (Kestrel, Kafka), or log aggregation system (Flume, Scribe) when something happens. Downstream <em>consumers</em> read messages from these systems, and process them or take actions based on the message contents.</p> @@ -165,7 +163,7 @@ <p>A messaging system lets you decouple all of this work from the actual web page serving.</p> -<h3 id="what-is-stream-processing?">What is stream processing?</h3> +<h3 id="what-is-stream-processing">What is stream processing?</h3> <p>A messaging system is a fairly low-level piece of infrastructure—it stores messages and waits for consumers to consume them. When you start writing code that produces or consumes messages, you quickly find that there are a lot of tricky problems that have to be solved in the processing layer. Samza aims to help with these problems.</p> @@ -202,7 +200,7 @@ <p>For a more in-depth discussion on Samza, and how it relates to other stream processing systems, have a look at Samza’s <a href="../comparisons/introduction.html">Comparisons</a> documentation.</p> -<h2 id="concepts-»"><a href="concepts.html">Concepts »</a></h2> +<h2 id="concepts"><a href="concepts.html">Concepts »</a></h2> </div> @@ -220,7 +218,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/background.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/background" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/concepts.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/concepts.html?rev=1832281&r1=1832280&r2=1832281&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/concepts.html (original) +++ samza/site/learn/documentation/latest/introduction/concepts.html Fri May 25 22:28:38 2018 @@ -44,9 +44,7 @@ <!-- this icon only shows in versioned pages --> - - - <a href="http://samza.apache.org/learn/documentation/0.14/introduction/concepts.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.14/introduction/concepts";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -197,7 +195,7 @@ <p>Partitions and tasks are both <em>logical</em> units of parallelism—they don’t correspond to any particular assignment of computational resources (CPU, memory, disk space, etc). Containers are the unit of physical parallelism, and a container is essentially a Unix process (or Linux <a href="http://en.wikipedia.org/wiki/Cgroups";>cgroup</a>). Each container runs one or more tasks. The number of tasks is determined automatically from the number of partitions in the input and is fixed, but the number of containers (and the CPU and memory resources associated with them) is specified by the user at run time and can be changed at any time.</p> -<h2 id="architecture-»"><a href="architecture.html">Architecture »</a></h2> +<h2 id="architecture"><a href="architecture.html">Architecture »</a></h2> </div> @@ -215,7 +213,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/concepts.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.14/introduction/concepts" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } });
