Modified: samza/site/startup/preview/index.html URL: http://svn.apache.org/viewvc/samza/site/startup/preview/index.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/startup/preview/index.html (original) +++ samza/site/startup/preview/index.html Wed Jan 18 19:33:25 2023 @@ -81,7 +81,7 @@ </div> <div class="main-navigation__items" data-menu-opened> <a class="main-navigation__item" href="/">Home</a> - <a class="main-navigation__item" href="/learn/documentation/1.8.0/core-concepts/core-concepts.html">Docs</a> + <a class="main-navigation__item" href="/learn/documentation/latest/core-concepts/core-concepts.html">Docs</a> <a class="main-navigation__item" href="/powered-by/">Powered By</a> <a class="main-navigation__item" href="/startup/download/">Downloads</a> <a class="main-navigation__item" href="/blog/">Blog</a> @@ -152,10 +152,10 @@ - <a class="side-navigation__group-item" data-match-active="" href="/startup/quick-start/1.8.0/">QuickStart</a> + <a class="side-navigation__group-item" data-match-active="" href="/startup/quick-start/latest/">QuickStart</a> - <a class="side-navigation__group-item" data-match-active="" href="/startup/code-examples/1.8.0/">Code Examples</a> + <a class="side-navigation__group-item" data-match-active="" href="/startup/code-examples/latest/">Code Examples</a> @@ -187,7 +187,7 @@ <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Documentation </div> - <div class="side-navigation__group-items side-navigation__group-has-submenus" data-sub-menu data-documentation="/learn/documentation/1.8.0/"> + <div class="side-navigation__group-items side-navigation__group-has-submenus" data-sub-menu data-documentation="/learn/documentation/latest/"> <!-- Handle sub navigation items from data --> @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -563,9 +569,9 @@ <p>Want to skip all the details and get some hands on experience? There are three tutorials to help you get acquainted with running Samza applications in both YARN and embedded modes and programming with the high level API:</p> <ul> - <li><a href="/learn/tutorials/1.8.0/hello-samza-high-level-yarn.html">Yarn Deployment</a> - run a pre-existing Wikipedia application on YARN and observe the output.</li> - <li><a href="/learn/tutorials/1.8.0/hello-samza-high-level-code.html">High Level API Code Walkthrough</a> - walk through building the Wikipedia application, step by step.</li> - <li><a href="/learn/tutorials/1.8.0/hello-samza-high-level-zk.html">ZooKeeper Deployment</a> - run a pre-existing Wikipedia application with ZooKeeper coordination and observe the output.</li> + <li><a href="/learn/tutorials/latest/hello-samza-high-level-yarn.html">Yarn Deployment</a> - run a pre-existing Wikipedia application on YARN and observe the output.</li> + <li><a href="/learn/tutorials/latest/hello-samza-high-level-code.html">High Level API Code Walkthrough</a> - walk through building the Wikipedia application, step by step.</li> + <li><a href="/learn/tutorials/latest/hello-samza-high-level-zk.html">ZooKeeper Deployment</a> - run a pre-existing Wikipedia application with ZooKeeper coordination and observe the output.</li> </ul> <hr /> @@ -578,19 +584,19 @@ <h3 id="concepts">Concepts</h3> <p>The Samza architecture has been overhauled with distinct layers to handle each stage of application development. The following diagram shows an overview of Apache Samza architecture with the high level API.</p> -<p><img src="/img/1.8.0/learn/documentation/introduction/layered-arch.png" alt="Architecture diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> +<p><img src="/img/latest/learn/documentation/introduction/layered-arch.png" alt="Architecture diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> <p>There are four layers in the architecture. The following sections describe each of the layers.</p> <h4 id="i-high-level-api">I. High Level API</h4> -<p>The high level API provides the libraries to define your application logic. The <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> is the central abstraction which your application must implement. You start by declaring your inputs as instances of <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/MessageStream.html">MessageStream</a>. Then you can apply operators on each MessageStream like map, filter, window, and join to define the whole end-to-end data processing in a single program.</p> +<p>The high level API provides the libraries to define your application logic. The <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> is the central abstraction which your application must implement. You start by declaring your inputs as instances of <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html">MessageStream</a>. Then you can apply operators on each MessageStream like map, filter, window, and join to define the whole end-to-end data processing in a single program.</p> <p>For a deeper dive into the high level API, see <a href="#high-level-api">high level API section</a> below.</p> <h4 id="ii-applicationrunner">II. ApplicationRunner</h4> -<p>Samza uses an <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/runtime/ApplicationRunner.html">ApplicationRunner</a> to run a stream application. The ApplicationRunner generates the configs (such as input/output streams), creates intermediate streams, and starts the execution. There are two types of ApplicationRunner:</p> +<p>Samza uses an <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/runtime/ApplicationRunner.html">ApplicationRunner</a> to run a stream application. The ApplicationRunner generates the configs (such as input/output streams), creates intermediate streams, and starts the execution. There are two types of ApplicationRunner:</p> <p><strong>RemoteApplicationRunner</strong> - submits the application to a remote cluster. This runner is invoked via the <em>run-app.sh</em> script. To use RemoteApplicationRunner, set the following configurations</p> @@ -598,7 +604,7 @@ app.class=com.company.job.YourStreamApplication job.factory.class=org.apache.samza.job.yarn.YarnJobFactory</code></pre></figure> -<p>Then use <em>run-app.sh</em> to run the application in the remote cluster. The script will invoke the RemoteApplicationRunner, which will launch one or more jobs using the factory specified with <em>job.factory.class</em>. Follow the <a href="/learn/tutorials/1.8.0/hello-samza-high-level-yarn.html">yarn deployment tutorial</a> to try it out.</p> +<p>Then use <em>run-app.sh</em> to run the application in the remote cluster. The script will invoke the RemoteApplicationRunner, which will launch one or more jobs using the factory specified with <em>job.factory.class</em>. Follow the <a href="/learn/tutorials/latest/hello-samza-high-level-yarn.html">yarn deployment tutorial</a> to try it out.</p> <p><strong>LocalApplicationRunner</strong> - runs the application in the JVM process of the runner. For example, to launch your application on multiple machines using ZooKeeper for coordination, you can run multiple instances of LocalApplicationRunner on various machines. After the applications load they will start cordinatinating their actions through ZooKeeper. Here is an example to run the StreamApplication in your program using the LocalApplicationRunner:</p> @@ -614,7 +620,7 @@ job.factory.class=org.apache.samza.job.y <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Application completed with status "</span> <span class="o">+</span> <span class="n">localRunner</span><span class="o">.</span><span class="na">status</span><span class="o">(</span><span class="n">app</span><span class="o">));</span> <span class="o">}</span></code></pre></figure> -<p>Follow the <a href="/learn/tutorials/1.8.0/hello-samza-high-level-zk.html">ZooKeeper deployment tutorial</a> to try it out.</p> +<p>Follow the <a href="/learn/tutorials/latest/hello-samza-high-level-zk.html">ZooKeeper deployment tutorial</a> to try it out.</p> <h5 id="execution-plan">Execution Plan</h5> @@ -627,7 +633,7 @@ job.factory.class=org.apache.samza.job.y <p>To view the plan, open the <em>bin/plan.html</em> file in a browser. Hereâs a sample plan visualization:</p> -<p><img src="/img/1.8.0/learn/documentation/introduction/execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> +<p><img src="/img/latest/learn/documentation/introduction/execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> <h4 id="iii-execution-models">III. Execution Models</h4> @@ -646,7 +652,7 @@ job.factory.class=org.apache.samza.job.y <h4 id="iv-processor">IV. Processor</h4> -<p>The lowest execution unit of a Samza application is the processor. It reads the configs generated from the <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/runtime/ApplicationRunner.html">ApplicationRunner</a> and processes the input stream partitions assigned by the JobCoordinator. It can access local state using a <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/storage/kv/KeyValueStore.html">KeyValueStore</a> implementation (e.g. RocksDB or in-memory) and remote state (e.g. REST service) using multithreading.</p> +<p>The lowest execution unit of a Samza application is the processor. It reads the configs generated from the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/runtime/ApplicationRunner.html">ApplicationRunner</a> and processes the input stream partitions assigned by the JobCoordinator. It can access local state using a <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/storage/kv/KeyValueStore.html">KeyValueStore</a> implementation (e.g. RocksDB or in-memory) and remote state (e.g. REST service) using multithreading.</p> <hr /> @@ -667,7 +673,7 @@ job.factory.class=org.apache.samza.job.y <h2 id="key-concepts">Key Concepts</h2> <h3 id="streamapplication">StreamApplication</h3> -<p>When writing your stream processing application using the Samza high-level API, you should implement a <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> and define your processing logic in the init method.</p> +<p>When writing your stream processing application using the Samza high-level API, you should implement a <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> and define your processing logic in the init method.</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="nc">StreamGraph</span> <span class="n">graph</span><span class="o">,</span> <span class="nc">Config</span> <span class="n">config</span><span class="o">)</span> <span class="o">{</span> <span class="err">â¦</span> <span class="o">}</span></code></pre></figure> @@ -738,17 +744,17 @@ streams.decorated-page-views.samza.physi <p>The second parameter is a serde to de-serialize the outgoing message.</p> <h2 id="operators">Operators</h2> -<p>The high level API supports common operators like map, flatmap, filter, merge, joins, and windowing on streams. Most of these operators accept corresponding Functions and these functions are <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/InitableFunction.html">Initable</a>.</p> +<p>The high level API supports common operators like map, flatmap, filter, merge, joins, and windowing on streams. Most of these operators accept corresponding Functions and these functions are <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/InitableFunction.html">Initable</a>.</p> <h3 id="map">Map</h3> -<p>Applies the provided 1:1 <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/MapFunction.html">MapFunction</a> to each element in the MessageStream and returns the transformed MessageStream. The MapFunction takes in a single message and returns a single message (potentially of a different type).</p> +<p>Applies the provided 1:1 <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/MapFunction.html">MapFunction</a> to each element in the MessageStream and returns the transformed MessageStream. The MapFunction takes in a single message and returns a single message (potentially of a different type).</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">MessageStream</span><span class="o"><</span><span class="nc">Integer</span><span class="o">></span> <span class="n">numbers</span> <span class="o">=</span> <span class="o">...</span> <span class="nc">MessageStream</span><span class="o"><</span><span class="nc">Integer</span><span class="o">></span> <span class="n">tripled</span><span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">m</span> <span class="o">-></span> <span class="n">m</span> <span class="o">*</span> <span class="mi">3</span><span class="o">)</span> <span class="nc">MessageStream</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">stringified</span> <span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">m</span> <span class="o">-></span> <span class="nc">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">m</span><span class="o">))</span></code></pre></figure> <h3 id="flatmap">Flatmap</h3> -<p>Applies the provided 1:n <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/FlatMapFunction.html">FlatMapFunction</a> to each element in the MessageStream and returns the transformed MessageStream. The FlatMapFunction takes in a single message and returns zero or more messages.</p> +<p>Applies the provided 1:n <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/FlatMapFunction.html">FlatMapFunction</a> to each element in the MessageStream and returns the transformed MessageStream. The FlatMapFunction takes in a single message and returns zero or more messages.</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">MessageStream</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">sentence</span> <span class="o">=</span> <span class="o">...</span> <span class="c1">// Parse the sentence into its individual words splitting by space</span> @@ -756,7 +762,7 @@ streams.decorated-page-views.samza.physi <span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">sentence</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="err">â</span> <span class="err">â</span><span class="o">))</span></code></pre></figure> <h3 id="filter">Filter</h3> -<p>Applies the provided <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/FilterFunction.html">FilterFunction</a> to the MessageStream and returns the filtered MessageStream. The FilterFunction is a predicate that specifies whether a message should be retained in the filtered stream. Messages for which the FilterFunction returns false are filtered out.</p> +<p>Applies the provided <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/FilterFunction.html">FilterFunction</a> to the MessageStream and returns the filtered MessageStream. The FilterFunction is a predicate that specifies whether a message should be retained in the filtered stream. Messages for which the FilterFunction returns false are filtered out.</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">MessageStream</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">words</span> <span class="o">=</span> <span class="o">...</span> <span class="c1">// Extract only the long words</span> @@ -787,7 +793,7 @@ streams.decorated-page-views.samza.physi <p>The merge transform preserves the order of each MessageStream, so if message <code class="language-plaintext highlighter-rouge">m1</code> appears before <code class="language-plaintext highlighter-rouge">m2</code> in any provided stream, then, <code class="language-plaintext highlighter-rouge">m1</code> also appears before <code class="language-plaintext highlighter-rouge">m2</code> in the merged stream.</p> -<p>As an alternative to the <code class="language-plaintext highlighter-rouge">merge</code> instance method, you also can use the <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/MessageStream.html#mergeAll-java.util.Collection-">MessageStream#mergeAll</a> static method to merge MessageStreams without operating on an initial stream.</p> +<p>As an alternative to the <code class="language-plaintext highlighter-rouge">merge</code> instance method, you also can use the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#mergeAll-java.util.Collection-">MessageStream#mergeAll</a> static method to merge MessageStreams without operating on an initial stream.</p> <h3 id="sendto-stream">SendTo (stream)</h3> <p>Sends all messages from this MessageStream to the provided OutputStream. You can specify the key and the value to be used for the outgoing message.</p> @@ -810,7 +816,7 @@ streams.decorated-page-views.samza.physi <span class="o">.</span><span class="na">sendTo</span><span class="o">(</span><span class="n">table</span><span class="o">);</span></code></pre></figure> <h3 id="sink">Sink</h3> -<p>Allows sending messages from this MessageStream to an output system using the provided <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/SinkFunction.html">SinkFunction</a>.</p> +<p>Allows sending messages from this MessageStream to an output system using the provided <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/SinkFunction.html">SinkFunction</a>.</p> <p>This offers more control than sendTo since the SinkFunction has access to the <code class="language-plaintext highlighter-rouge">MessageCollector</code> and the <code class="language-plaintext highlighter-rouge">TaskCoordinator</code>. For instance, you can choose to manually commit offsets, or shut-down the job using the TaskCoordinator APIs. This operator can also be used to send messages to non-Samza systems (e.g. remote databases, REST services, etc.)</p> @@ -824,7 +830,7 @@ streams.decorated-page-views.samza.physi <h3 id="join-stream-stream">Join (stream-stream)</h3> -<p>The stream-stream Join operator joins messages from two MessageStreams using the provided pairwise <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/JoinFunction.html">JoinFunction</a>. Messages are joined when the keys extracted from messages from the first stream match keys extracted from messages in the second stream. Messages in each stream are retained for the provided ttl duration and join results are emitted as matches are found.</p> +<p>The stream-stream Join operator joins messages from two MessageStreams using the provided pairwise <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/JoinFunction.html">JoinFunction</a>. Messages are joined when the keys extracted from messages from the first stream match keys extracted from messages in the second stream. Messages in each stream are retained for the provided ttl duration and join results are emitted as matches are found.</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// Joins a stream of OrderRecord with a stream of ShipmentRecord by orderId with a TTL of 20 minutes.</span> <span class="c1">// Results are produced to a new stream of FulfilledOrderRecord.</span> @@ -857,7 +863,7 @@ streams.decorated-page-views.samza.physi <h3 id="join-stream-table">Join (stream-table)</h3> -<p>The stream-table Join operator joins messages from a MessageStream using the provided <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/StreamTableJoinFunction.html">StreamTableJoinFunction</a>. Messages from the input stream are joined with record in table using key extracted from input messages. The join function is invoked with both the message and the record. If a record is not found in the table, a null value is provided; the join function can choose to return null (inner join) or an output message (left outer join). For join to function properly, it is important to ensure the input stream and table are partitioned using the same key as this impacts the physical placement of data.</p> +<p>The stream-table Join operator joins messages from a MessageStream using the provided <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/StreamTableJoinFunction.html">StreamTableJoinFunction</a>. Messages from the input stream are joined with record in table using key extracted from input messages. The join function is invoked with both the message and the record. If a record is not found in the table, a null value is provided; the join function can choose to return null (inner join) or an output message (left outer join). For join to function properly, it is important to ensure the input stream and table are partitioned using the same key as this impacts the physical placement of data.</p> <figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"PageView"</span><span class="o">,</span> <span class="k">new</span> <span class="nc">NoOpSerde</span><span class="o"><</span><span class="nc">PageView</span><span class="o">>())</span> <span class="o">.</span><span class="na">partitionBy</span><span class="o">(</span><span class="nl">PageView:</span><span class="o">:</span><span class="n">getMemberId</span><span class="o">,</span> <span class="n">v</span> <span class="o">-></span> <span class="n">v</span><span class="o">,</span> <span class="s">"p1"</span><span class="o">)</span> @@ -909,9 +915,9 @@ streams.decorated-page-views.samza.physi <h4 id="windowing-concepts">Windowing Concepts</h4> <p><strong>Windows, Triggers, and WindowPanes</strong>: The window operator groups incoming messages in the MessageStream into finite windows. Each emitted result contains one or more messages in the window and is called a WindowPane.</p> -<p>A window can have one or more associated triggers which determine when results from the window are emitted. Triggers can be either <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/windows/Window.html#setEarlyTrigger-org.apache.samza.operators.triggers.Trigger-">early triggers</a> that allow emitting results speculatively before all data for the window has arrived, or late triggers that allow handling late messages for the window.</p> +<p>A window can have one or more associated triggers which determine when results from the window are emitted. Triggers can be either <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/windows/Window.html#setEarlyTrigger-org.apache.samza.operators.triggers.Trigger-">early triggers</a> that allow emitting results speculatively before all data for the window has arrived, or late triggers that allow handling late messages for the window.</p> -<p><strong>Aggregator Function</strong>: By default, the emitted WindowPane will contain all the messages for the window. Instead of retaining all messages, you typically define a more compact data structure for the WindowPane and update it incrementally as new messages arrive, e.g. for keeping a count of messages in the window. To do this, you can provide an aggregating <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/operators/functions/FoldLeftFunction.html">FoldLeftFunction</a> which is invoked for each incoming message added to the window and defines how to update the WindowPane for that message.</p> +<p><strong>Aggregator Function</strong>: By default, the emitted WindowPane will contain all the messages for the window. Instead of retaining all messages, you typically define a more compact data structure for the WindowPane and update it incrementally as new messages arrive, e.g. for keeping a count of messages in the window. To do this, you can provide an aggregating <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/FoldLeftFunction.html">FoldLeftFunction</a> which is invoked for each incoming message added to the window and defines how to update the WindowPane for that message.</p> <p><strong>Accumulation Mode</strong>: A windowâs accumulation mode determines how results emitted from a window relate to previously emitted results for the same window. This is particularly useful when the window is configured with early or late triggers. The accumulation mode can either be discarding or accumulating.</p> @@ -1011,7 +1017,7 @@ streams.decorated-page-views.samza.physi <p>There are two types of partition assignment models which are controlled with the <em>job.coordinator.factory</em> in configuration:</p> <h4 id="external-partition-management">External Partition Management</h4> -<p>With external partition management, Samza doesnât manage the partitioning by itself. Instead it uses a <code class="language-plaintext highlighter-rouge">PassthroughJobCoordinator</code> which honors whatever partition mapping is provided by the <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/container/grouper/stream/SystemStreamPartitionGrouper.html">SystemStreamPartitionGrouper</a>. There are two common patterns for external partition management:</p> +<p>With external partition management, Samza doesnât manage the partitioning by itself. Instead it uses a <code class="language-plaintext highlighter-rouge">PassthroughJobCoordinator</code> which honors whatever partition mapping is provided by the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/container/grouper/stream/SystemStreamPartitionGrouper.html">SystemStreamPartitionGrouper</a>. There are two common patterns for external partition management:</p> <ul> <li><strong>Using high level Kafka consumer</strong> - partition assignment is done by the high level Kafka consumer itself. To use this model, you need to implement and configure a SystemFactory which provides the Kafka high level consumer. Then you need to configure <em>job.systemstreampartition.grouper.factory</em> to <em>org.apache.samza.container.grouper.stream.AllSspToSingleTaskGrouper</em> so Kafkaâs partition assignments all go to one task.</li> @@ -1055,7 +1061,7 @@ There is one leader processor which gene <p>The following diagram shows the relationships of the coordinators in the ZooKeeper coordination service implementation.</p> -<p><img src="/img/1.8.0/learn/documentation/introduction/coordination-service.png" alt="Coordination service diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> +<p><img src="/img/latest/learn/documentation/introduction/coordination-service.png" alt="Coordination service diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> <p>Here are a few important details about the coordination service:</p> @@ -1068,7 +1074,7 @@ There is one leader processor which gene <h3 id="user-guide">User guide</h3> <p>Embedded deployment is designed to help users who want more control over the deployment of their application. So it is the userâs responsibility to configure and deploy the processors. In case of ZooKeeper coordination, you also need to configure the URL for an instance of ZooKeeper.</p> -<p>Additionally, each processor requires a unique ID to be used with the coordination service. If location affinity is important, this ID should be unique for each processor on a specific hostname (assuming local Storage services). To address this requirement, Samza uses a <a href="/learn/documentation/1.8.0/api/javadocs/org/apache/samza/runtime/ProcessorIdGenerator.html">ProcessorIdGenerator</a> to provide the ID for each processor. If no generator is explicitly configured, the default one will create a UUID for each processor.</p> +<p>Additionally, each processor requires a unique ID to be used with the coordination service. If location affinity is important, this ID should be unique for each processor on a specific hostname (assuming local Storage services). To address this requirement, Samza uses a <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/runtime/ProcessorIdGenerator.html">ProcessorIdGenerator</a> to provide the ID for each processor. If no generator is explicitly configured, the default one will create a UUID for each processor.</p> <h4 id="configuration">Configuration</h4> <p>To run an embedded Samza processor, you need to configure the coordinator service using the <em>job.coordinator.factory</em> property. Also, there is currently one taskname grouper that supports embedded mode, so you must configure that explicitly.</p> @@ -1106,7 +1112,7 @@ task.name.grouper.factory=org.apache.sam <p>In the code above, <code class="language-plaintext highlighter-rouge">WikipediaApplication</code> is an application written with the <a href="#high-level-api">high level API</a>.</p> -<p>Check out the <a href="/learn/tutorials/1.8.0/hello-samza-high-level-zk.html">tutorial</a> to run this application with ZooKeeper coordination on your machine now.</p> +<p>Check out the <a href="/learn/tutorials/latest/hello-samza-high-level-zk.html">tutorial</a> to run this application with ZooKeeper coordination on your machine now.</p> <h4 id="deployment-and-scaling">Deployment and Scaling</h4> <p>You can deploy the application instances in any way you prefer. If using the coordination service, you can add or remove instances at any time and the leaderâs job coordinator (elected via the CoordinationService) will automatically recalculate the JobModel after the debounce time and apply it to the available processors. So, to scale up your application, you simply start more processors.</p> @@ -1163,7 +1169,7 @@ task.name.grouper.factory=org.apache.sam <div class="footer__items"> <a class="footer__item" href="/meetups/">Meetups</a> <a class="footer__item" href="/blog/">Blog</a> - <a class="footer__item" href="/learn/documentation/1.8.0/introduction/background.html">About</a> + <a class="footer__item" href="/learn/documentation/latest/introduction/background.html">About</a> </div> </div> <div>
Modified: samza/site/startup/quick-start/latest/beam.html URL: http://svn.apache.org/viewvc/samza/site/startup/quick-start/latest/beam.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/startup/quick-start/latest/beam.html (original) +++ samza/site/startup/quick-start/latest/beam.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -545,79 +551,79 @@ limitations under the License. --> -<p><a href="https://beam.apache.org/";>Apache Beam</a> is an open-source SDK which provides state-of-the-art data processing API and model for both batch and streaming processing pipelines across multiple languages, i.e. Java, Python and Go. By collaborating with Beam, Samza offers the capability of executing Beam API on Samza’s large-scale and stateful streaming engine. Current Samza supports the full Beam <a href="https://beam.apache.org/documentation/runners/capability-matrix/";>Java API</a>, and the support of Python and Go is work-in-progress.</p> +<p><a href="https://beam.apache.org/";>Apache Beam</a> is an open-source SDK which provides state-of-the-art data processing API and model for both batch and streaming processing pipelines across multiple languages, i.e. Java, Python and Go. By collaborating with Beam, Samza offers the capability of executing Beam API on Samzaâs large-scale and stateful streaming engine. Current Samza supports the full Beam <a href="https://beam.apache.org/documentation/runners/capability-matrix/";>Java API</a>, and the support of Python and Go is work-in-progress.</p> <h3 id="setting-up-the-word-count-project">Setting up the Word-Count Project</h3> <p>To get started, you need to install <a href="">Java 8 SDK</a> as well as <a href="http://maven.apache.org/download.cgi";>Apache Maven</a>. After that, the easiest way to get a copy of the WordCount examples in Beam API is to use the following command to generate a simple Maven project:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>> mvn archetype:generate <span class="se">\</span> - -DarchetypeGroupId<span class="o">=</span>org.apache.beam <span class="se">\</span> - -DarchetypeArtifactId<span class="o">=</span>beam-sdks-java-maven-archetypes-examples <span class="se">\</span> - -DarchetypeVersion<span class="o">=</span><span class="m">2</span>.11.0 <span class="se">\</span> - -DgroupId<span class="o">=</span>org.example <span class="se">\</span> - -DartifactId<span class="o">=</span>word-count-beam <span class="se">\</span> - -Dversion<span class="o">=</span><span class="s2">"0.1"</span> <span class="se">\</span> - -Dpackage<span class="o">=</span>org.apache.beam.examples <span class="se">\</span> - -DinteractiveMode<span class="o">=</span><span class="nb">false</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> mvn archetype:generate <span class="se">\</span> + <span class="nt">-DarchetypeGroupId</span><span class="o">=</span>org.apache.beam <span class="se">\</span> + <span class="nt">-DarchetypeArtifactId</span><span class="o">=</span>beam-sdks-java-maven-archetypes-examples <span class="se">\</span> + <span class="nt">-DarchetypeVersion</span><span class="o">=</span>2.11.0 <span class="se">\</span> + <span class="nt">-DgroupId</span><span class="o">=</span>org.example <span class="se">\</span> + <span class="nt">-DartifactId</span><span class="o">=</span>word-count-beam <span class="se">\</span> + <span class="nt">-Dversion</span><span class="o">=</span><span class="s2">"0.1"</span> <span class="se">\</span> + <span class="nt">-Dpackage</span><span class="o">=</span>org.apache.beam.examples <span class="se">\</span> + <span class="nt">-DinteractiveMode</span><span class="o">=</span><span class="nb">false</span></code></pre></figure> -<p>This command creates a maven project <code>word-count-beam</code> which contains a series of example pipelines that count words in text files:</p> +<p>This command creates a maven project <code class="language-plaintext highlighter-rouge">word-count-beam</code> which contains a series of example pipelines that count words in text files:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>> <span class="nb">cd</span> word-count-beam/ +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">cd </span>word-count-beam/ -> ls src/main/java/org/apache/beam/examples/ -DebuggingWordCount.java WindowedWordCount.java common -MinimalWordCount.java WordCount.java</code></pre></figure> +<span class="o">></span> <span class="nb">ls </span>src/main/java/org/apache/beam/examples/ +DebuggingWordCount.java WindowedWordCount.java common +MinimalWordCount.java WordCount.java</code></pre></figure> -<p>Let’s use the MinimalWordCount example to demonstrate how to create a simple Beam pipeline:</p> +<p>Letâs use the MinimalWordCount example to demonstrate how to create a simple Beam pipeline:</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinimalWordCount</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinimalWordCount</span> <span class="o">{</span> - <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// Create the Pipeline object with the options we defined above</span> - <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> - <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> + <span class="nc">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="nc">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> + <span class="nc">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="nc">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> <span class="c1">// This example reads a public data set consisting of the complete works of Shakespeare.</span> - <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">"gs://apache-beam-samples/shakespeare/*"</span><span class="o">))</span> + <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="nc">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">"gs://apache-beam-samples/shakespeare/*"</span><span class="o">))</span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> - <span class="n">FlatMapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> - <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-></span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">"[^\\p{L}]+"</span><span class="o">))))</span> - <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">((</span><span class="n">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-></span> <span class="o">!</span><span class="n">word</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()))</span> - <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Count</span><span class="o">.</span><span class="na">perElement</span><span class="o">())</span> + <span class="nc">FlatMapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="nc">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> + <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="nc">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-></span> <span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">"[^\\p{L}]+"</span><span class="o">))))</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="nc">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">((</span><span class="nc">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-></span> <span class="o">!</span><span class="n">word</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()))</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="nc">Count</span><span class="o">.</span><span class="na">perElement</span><span class="o">())</span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> - <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> + <span class="nc">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="nc">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> <span class="o">.</span><span class="na">via</span><span class="o">(</span> - <span class="o">(</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="n">wordCount</span><span class="o">)</span> <span class="o">-></span> - <span class="n">wordCount</span><span class="o">.</span><span class="na">getKey</span><span class="o">()</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="n">wordCount</span><span class="o">.</span><span class="na">getValue</span><span class="o">()))</span> - <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">"wordcounts"</span><span class="o">));</span> + <span class="o">(</span><span class="no">KV</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Long</span><span class="o">></span> <span class="n">wordCount</span><span class="o">)</span> <span class="o">-></span> + <span class="n">wordCount</span><span class="o">.</span><span class="na">getKey</span><span class="o">()</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="n">wordCount</span><span class="o">.</span><span class="na">getValue</span><span class="o">()))</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="nc">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">"wordcounts"</span><span class="o">));</span> <span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span> <span class="o">}</span> <span class="o">}</span></code></pre></figure> -<p>In this example, we first create a Beam <code>Pipeline</code> object to build the graph of transformations to be executed. Then we first use the Read transform to consume a public data set, and split into words. Then we use Beam build-in <code>Count</code> transform and returns the key/value pairs where each key represents a unique element from the input collection, and each value represents the number of times that key appeared in the input collection. Finally we format the results and write them to a file. A detailed walkthrough of the example code can be found <a href="https://beam.apache.org/get-started/wordcount-example/";>here</a>.</p> +<p>In this example, we first create a Beam <code class="language-plaintext highlighter-rouge">Pipeline</code> object to build the graph of transformations to be executed. Then we first use the Read transform to consume a public data set, and split into words. Then we use Beam build-in <code class="language-plaintext highlighter-rouge">Count</code> transform and returns the key/value pairs where each key represents a unique element from the input collection, and each value represents the number of times that key appeared in the input collection. Finally we format the results and write them to a file. A detailed walkthrough of the example code can be found <a href="https://beam.apache.org/get-started/wordcount-example/";>here</a>.</p> -<p>Let’s run the WordCount example with Samza using the following command:</p> +<p>Letâs run the WordCount example with Samza using the following command:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>>mvn compile exec:java -Dexec.mainClass<span class="o">=</span>org.apache.beam.examples.WordCount <span class="se">\</span> - -Dexec.args<span class="o">=</span><span class="s2">"--inputFile=pom.xml --output=/tmp/counts --runner=SamzaRunner"</span> -Psamza-runner</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span>mvn compile <span class="nb">exec</span>:java <span class="nt">-Dexec</span>.mainClass<span class="o">=</span>org.apache.beam.examples.WordCount <span class="se">\</span> + <span class="nt">-Dexec</span>.args<span class="o">=</span><span class="s2">"--inputFile=pom.xml --output=/tmp/counts --runner=SamzaRunner"</span> <span class="nt">-Psamza-runner</span></code></pre></figure> <p>After the pipeline finishes, you can check out the output counts files in /tmp folder. Note Beam generates multiple output files for parallel processing. If you prefer a single output, please update the code to use TextIO.write().withoutSharding().</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>>more /tmp/counts* -This: <span class="m">1</span> -When: <span class="m">1</span> -YARN: <span class="m">1</span> -apex: <span class="m">2</span> -apis: <span class="m">2</span> -beam: <span class="m">43</span> -beta: <span class="m">1</span> -code: <span class="m">2</span> -copy: <span class="m">1</span> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span>more /tmp/counts<span class="k">*</span> +This: 1 +When: 1 +YARN: 1 +apex: 2 +apis: 2 +beam: 43 +beta: 1 +code: 2 +copy: 1 ...</code></pre></figure> -<p>For more examples and how to deploy your job in local, standalone and Yarn cluster, you can look at the <a href="/startup/code-examples/latest/beam.html">code examples</a>. Please don’t hesitate to <a href="https://samza.apache.org/community/contact-us.html";>reach out</a> if you encounter any issues.</p> +<p>For more examples and how to deploy your job in local, standalone and Yarn cluster, you can look at the <a href="/startup/code-examples/latest/beam.html">code examples</a>. Please donât hesitate to <a href="https://samza.apache.org/community/contact-us.html";>reach out</a> if you encounter any issues.</p> </div> Modified: samza/site/startup/quick-start/latest/index.html URL: http://svn.apache.org/viewvc/samza/site/startup/quick-start/latest/index.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/startup/quick-start/latest/index.html (original) +++ samza/site/startup/quick-start/latest/index.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> Modified: samza/site/startup/quick-start/latest/samza-sql.html URL: http://svn.apache.org/viewvc/samza/site/startup/quick-start/latest/samza-sql.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/startup/quick-start/latest/samza-sql.html (original) +++ samza/site/startup/quick-start/latest/samza-sql.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -546,47 +552,55 @@ --> <h3 id="overview">Overview</h3> - <p>Samza SQL allows you to define your stream processing logic declaratively as a a SQL query. This allows you to create streaming pipelines without Java code or configuration unless you -require user-defined functions (<a href="#how-to-write-a-udf">UDF</a>). </p> +require user-defined functions (<a href="#how-to-write-a-udf">UDF</a>).</p> <p>You can run Samza SQL locally on your machine or on a YARN cluster.</p> <h3 id="running-samza-sql-on-your-local-machine">Running Samza SQL on your local machine</h3> - -<p>The <a href="https://samza.apache.org/learn/tutorials/0.14/samza-tools.html";>Samza SQL console</a> allows you to experiment with Samza SQL locally on your machine. </p> +<p>The <a href="https://samza.apache.org/learn/tutorials/0.14/samza-tools.html";>Samza SQL console</a> allows you to experiment with Samza SQL locally on your machine.</p> <h4 id="setup-kafka">Setup Kafka</h4> - <p>Follow the instructions from the <a href="http://kafka.apache.org/quickstart";>Kafka quickstart</a> to start the zookeeper and Kafka server.</p> <p>Let us create a Kafka topic named âProfileChangeStreamâ for this demo.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor <span class="m">1</span> --partitions <span class="m">1</span> --topic ProfileChangeStream -</code></pre></div> -<p>Download the Samza tools package from <a href="https://samza.apache.org/learn/tutorials/0.14/samza-tools.html";>here</a> and use the <code>generate-kafka-events</code> script populate the stream with sample data.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> samza-tools-<version> -./scripts/generate-kafka-events.sh -t ProfileChangeStream -e ProfileChange -</code></pre></div> + +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./deploy/kafka/bin/kafka-topics.sh <span class="nt">--create</span> <span class="nt">--zookeeper</span> localhost:2181 <span class="nt">--replication-factor</span> 1 <span class="nt">--partitions</span> 1 <span class="nt">--topic</span> ProfileChangeStream +</code></pre></div></div> + +<p>Download the Samza tools package from <a href="https://samza.apache.org/learn/tutorials/0.14/samza-tools.html";>here</a> and use the <code class="language-plaintext highlighter-rouge">generate-kafka-events</code> script populate the stream with sample data.</p> + +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>samza-tools-<version> +./scripts/generate-kafka-events.sh <span class="nt">-t</span> ProfileChangeStream <span class="nt">-e</span> ProfileChange +</code></pre></div></div> + <h4 id="using-the-samza-sql-console">Using the Samza SQL Console</h4> -<p>The simplest SQL query is to read all events from a Kafka topic <code>ProfileChangeStream</code> and print them to the console.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./scripts/samza-sql-console.sh --sql <span class="s2">"insert into log.consoleoutput select * from kafka.ProfileChangeStream"</span> -</code></pre></div> +<p>The simplest SQL query is to read all events from a Kafka topic <code class="language-plaintext highlighter-rouge">ProfileChangeStream</code> and print them to the console.</p> + +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./scripts/samza-sql-console.sh <span class="nt">--sql</span> <span class="s2">"insert into log.consoleoutput select * from kafka.ProfileChangeStream"</span> +</code></pre></div></div> + <p>Next, let us project a few fields from the input stream.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./scripts/samza-sql-console.sh --sql <span class="s2">"insert into log.consoleoutput select Name, OldCompany, NewCompany from kafka.ProfileChangeStream"</span> -</code></pre></div> -<p>You can also filter messages in the input stream based on some predicate. In this example, we filter profiles currently working at LinkedIn, whose previous employer matches the regex <code>.*soft</code>. The function <code>RegexMatch(regex, company)</code> is an example of -a UDF that defines a predicate. </p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./scripts/samza-sql-console.sh --sql <span class="s2">"insert into log.consoleoutput select Name as __key__, Name, NewCompany, RegexMatch('.*soft', OldCompany) from kafka.ProfileChangeStream where NewCompany = 'LinkedIn'"</span> -</code></pre></div> -<h3 id="running-samza-sql-on-yarn">Running Samza SQL on YARN</h3> +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./scripts/samza-sql-console.sh <span class="nt">--sql</span> <span class="s2">"insert into log.consoleoutput select Name, OldCompany, NewCompany from kafka.ProfileChangeStream"</span> +</code></pre></div></div> + +<p>You can also filter messages in the input stream based on some predicate. In this example, we filter profiles currently working at LinkedIn, whose previous employer matches the regex <code class="language-plaintext highlighter-rouge">.*soft</code>. The function <code class="language-plaintext highlighter-rouge">RegexMatch(regex, company)</code> is an example of +a UDF that defines a predicate.</p> + +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./scripts/samza-sql-console.sh <span class="nt">--sql</span> <span class="s2">"insert into log.consoleoutput select Name as __key__, Name, NewCompany, RegexMatch('.*soft', OldCompany) from kafka.ProfileChangeStream where NewCompany = 'LinkedIn'"</span> +</code></pre></div></div> + +<h3 id="running-samza-sql-on-yarn">Running Samza SQL on YARN</h3> <p>The <a href="https://github.com/apache/samza-hello-samza";>hello-samza</a> project has examples to get started with Samza on YARN. You can define your SQL query in a configuration file and submit it to a YARN cluster.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/page-view-filter-sql.properties -</code></pre></div> + +<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/page-view-filter-sql.properties +</code></pre></div></div> + <h3 id="how-to-write-a-udf">How to write a UDF</h3> <p>Right now Samza SQL support Scalar UDFs which means that each @@ -595,32 +609,33 @@ configuration file and submit it to a YA of 1 output to an input. Users need to implement the following interface to create a UDF.</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="cm">/**</span> -<span class="cm"> * The base class for the Scalar UDFs. All the scalar UDF classes needs to extend this and implement a method named</span> -<span class="cm"> * "execute". The number of arguments for the execute method in the UDF class should match the number of fields</span> -<span class="cm"> * used while invoking this UDF in SQL statement.</span> -<span class="cm"> * Say for e.g. User creates a UDF class with signature int execute(int var1, String var2). It can be used in a SQL query</span> -<span class="cm"> * select myudf(id, name) from profile</span> -<span class="cm"> * In the above query, Profile should contain fields named 'id' of INTEGER/NUMBER type and 'name' of type VARCHAR/CHARACTER</span> -<span class="cm"> */</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="cm">/** + * The base class for the Scalar UDFs. All the scalar UDF classes needs to extend this and implement a method named + * "execute". The number of arguments for the execute method in the UDF class should match the number of fields + * used while invoking this UDF in SQL statement. + * Say for e.g. User creates a UDF class with signature int execute(int var1, String var2). It can be used in a SQL query + * select myudf(id, name) from profile + * In the above query, Profile should contain fields named 'id' of INTEGER/NUMBER type and 'name' of type VARCHAR/CHARACTER + */</span> <span class="kd">public</span> <span class="kd">interface</span> <span class="nc">ScalarUdf</span> <span class="o">{</span> - <span class="cm">/**</span> -<span class="cm"> * Udfs can implement this method to perform any initialization that they may need.</span> -<span class="cm"> * @param udfConfig Config specific to the udf.</span> -<span class="cm"> */</span> - <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">udfConfig</span><span class="o">);</span> + <span class="cm">/** + * Udfs can implement this method to perform any initialization that they may need. + * @param udfConfig Config specific to the udf. + */</span> + <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="nc">Config</span> <span class="n">udfConfig</span><span class="o">);</span> - <span class="cm">/**</span> -<span class="cm"> * Actual implementation of the udf function</span> -<span class="cm"> * @param args</span> -<span class="cm"> * list of all arguments that the udf needs</span> -<span class="cm"> * @return</span> -<span class="cm"> * Return value from the scalar udf.</span> -<span class="cm"> */</span> - <span class="n">Object</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Object</span><span class="o">...</span> <span class="n">args</span><span class="o">);</span> + <span class="cm">/** + * Actual implementation of the udf function + * @param args + * list of all arguments that the udf needs + * @return + * Return value from the scalar udf. + */</span> + <span class="nc">Object</span> <span class="nf">execute</span><span class="o">(</span><span class="nc">Object</span><span class="o">...</span> <span class="n">args</span><span class="o">);</span> <span class="o">}</span> </code></pre></figure> + </div> </div>
