Modified: samza/site/learn/documentation/latest/api/programming-model.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/api/programming-model.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/api/programming-model.html (original) +++ samza/site/learn/documentation/latest/api/programming-model.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/api/programming-model">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/api/programming-model">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/api/programming-model">1.6.0</a></li> @@ -638,79 +652,74 @@ See the License for the specific language governing permissions and limitations under the License. --> - <h3 id="introduction">Introduction</h3> - <p>Samza provides multiple programming APIs to fit your use case:</p> <ol> -<li>Java APIs: Samza’s provides two Java programming APIs that are ideal for building advanced Stream Processing applications. - -<ol> -<li><a href="high-level-api.html">High Level Streams API</a>: Samza’s flexible High Level Streams API lets you describe your complex stream processing pipeline in the form of a Directional Acyclic Graph (DAG) of operations on message streams. It provides a rich set of built-in operators that simplify common stream processing operations such as filtering, projection, repartitioning, joins, and windows.</li> -<li><a href="low-level-api.html">Low Level Task API</a>: Samza’s powerful Low Level Task API lets you write your application in terms of processing logic for each incoming message. </li> -</ol></li> -<li><a href="samza-sql.html">Samza SQL</a>: Samza SQL provides a declarative query language for describing your stream processing logic. It lets you manipulate streams using SQL predicates and UDFs instead of working with the physical implementation details.</li> -<li>Apache Beam API: Samza also provides a <a href="https://beam.apache.org/documentation/runners/capability-matrix/";>Apache Beam runner</a> to run applications written using the Apache Beam API. This is considered as an extension to the operators supported by the High Level Streams API in Samza.</li> + <li>Java APIs: Samzaâs provides two Java programming APIs that are ideal for building advanced Stream Processing applications. + <ol> + <li><a href="high-level-api.html">High Level Streams API</a>: Samzaâs flexible High Level Streams API lets you describe your complex stream processing pipeline in the form of a Directional Acyclic Graph (DAG) of operations on message streams. It provides a rich set of built-in operators that simplify common stream processing operations such as filtering, projection, repartitioning, joins, and windows.</li> + <li><a href="low-level-api.html">Low Level Task API</a>: Samzaâs powerful Low Level Task API lets you write your application in terms of processing logic for each incoming message.</li> + </ol> + </li> + <li><a href="samza-sql.html">Samza SQL</a>: Samza SQL provides a declarative query language for describing your stream processing logic. It lets you manipulate streams using SQL predicates and UDFs instead of working with the physical implementation details.</li> + <li>Apache Beam API: Samza also provides a <a href="https://beam.apache.org/documentation/runners/capability-matrix/";>Apache Beam runner</a> to run applications written using the Apache Beam API. This is considered as an extension to the operators supported by the High Level Streams API in Samza.</li> </ol> <h3 id="key-concepts">Key Concepts</h3> - <p>The following sections will talk about key concepts in writing your Samza applications in Java.</p> <h4 id="samza-applications">Samza Applications</h4> - -<p>A <a href="javadocs/org/apache/samza/application/SamzaApplication.html">SamzaApplication</a> describes the inputs, outputs, state, configuration and the logic for processing data from one or more streaming sources. </p> +<p>A <a href="javadocs/org/apache/samza/application/SamzaApplication.html">SamzaApplication</a> describes the inputs, outputs, state, configuration and the logic for processing data from one or more streaming sources.</p> <p>You can implement a -<a href="javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> and use the provided <a href="javadocs/org/apache/samza/application/descriptors/StreamApplicationDescriptor">StreamApplicationDescriptor</a> to describe the processing logic using Samza’s High Level Streams API in terms of <a href="javadocs/org/apache/samza/operators/MessageStream.html">MessageStream</a> operators. </p> +<a href="javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> and use the provided <a href="javadocs/org/apache/samza/application/descriptors/StreamApplicationDescriptor">StreamApplicationDescriptor</a> to describe the processing logic using Samzaâs High Level Streams API in terms of <a href="javadocs/org/apache/samza/operators/MessageStream.html">MessageStream</a> operators.</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStreamApplication</span> <span class="kd">implements</span> <span class="n">StreamApplication</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStreamApplication</span> <span class="kd">implements</span> <span class="nc">StreamApplication</span> <span class="o">{</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="n">StreamApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="nc">StreamApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// Describe your application here </span> <span class="o">}</span> <span class="o">}</span></code></pre></figure> -<p>Alternatively, you can implement a <a href="javadocs/org/apache/samza/application/TaskApplication.html">TaskApplication</a> and use the provided <a href="javadocs/org/apache/samza/application/descriptors/TaskApplicationDescriptor">TaskApplicationDescriptor</a> to describe it using Samza’s Low Level API in terms of per-message processing logic.</p> +<p>Alternatively, you can implement a <a href="javadocs/org/apache/samza/application/TaskApplication.html">TaskApplication</a> and use the provided <a href="javadocs/org/apache/samza/application/descriptors/TaskApplicationDescriptor">TaskApplicationDescriptor</a> to describe it using Samzaâs Low Level API in terms of per-message processing logic.</p> <ul> -<li>For TaskApplication:</li> + <li>For TaskApplication:</li> </ul> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> - <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyTaskApplication</span> <span class="kd">implements</span> <span class="n">TaskApplication</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> + <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyTaskApplication</span> <span class="kd">implements</span> <span class="nc">TaskApplication</span> <span class="o">{</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="n">TaskApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="nc">TaskApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// Describe your application here</span> <span class="o">}</span> <span class="o">}</span></code></pre></figure> <h4 id="streams-and-table-descriptors">Streams and Table Descriptors</h4> +<p>Descriptors let you specify the properties of various aspects of your application from within it.</p> -<p>Descriptors let you specify the properties of various aspects of your application from within it. </p> - -<p><a href="javadocs/org/apache/samza/system/descriptors/InputDescriptor.html">InputDescriptor</a>s and <a href="javadocs/org/apache/samza/system/descriptors/OutputDescriptor.html">OutputDescriptor</a>s can be used for specifying Samza and implementation-specific properties of the streaming inputs and outputs for your application. You can obtain InputDescriptors and OutputDescriptors using a <a href="javadocs/org/apache/samza/system/descriptors/SystemDescriptor.html">SystemDescriptor</a> for your system. This SystemDescriptor can be used for specify Samza and implementation-specific properties of the producer and consumers for your I/O system. Most Samza system implementations come with their own SystemDescriptors, but if one isn’t available, you +<p><a href="javadocs/org/apache/samza/system/descriptors/InputDescriptor.html">InputDescriptor</a>s and <a href="javadocs/org/apache/samza/system/descriptors/OutputDescriptor.html">OutputDescriptor</a>s can be used for specifying Samza and implementation-specific properties of the streaming inputs and outputs for your application. You can obtain InputDescriptors and OutputDescriptors using a <a href="javadocs/org/apache/samza/system/descriptors/SystemDescriptor.html">SystemDescriptor</a> for your system. This SystemDescriptor can be used for specify Samza and implementation-specific properties of the producer and consumers for your I/O system. Most Samza system implementations come with their own SystemDescriptors, but if one isnât available, you can use the <a href="javadocs/org/apache/samza/system/descriptors/GenericSystemDescriptor.html">GenericSystemDescriptor</a>.</p> -<p>A <a href="javadocs/org/apache/samza/table/descriptors/TableDescriptor.html">TableDescriptor</a> can be used for specifying Samza and implementation-specific properties of a <a href="javadocs/org/apache/samza/table/Table.html">Table</a>. You can use a Local TableDescriptor (e.g. <a href="javadocs/org/apache/samza/storage/kv/descriptors/RocksDbTableDescriptor.html">RocksDbTableDescriptor</a> or a <a href="javadocs/org/apache/samza/table/descriptors/RemoteTableDescriptor">RemoteTableDescriptor</a>. </p> +<p>A <a href="javadocs/org/apache/samza/table/descriptors/TableDescriptor.html">TableDescriptor</a> can be used for specifying Samza and implementation-specific properties of a <a href="javadocs/org/apache/samza/table/Table.html">Table</a>. You can use a Local TableDescriptor (e.g. <a href="javadocs/org/apache/samza/storage/kv/descriptors/RocksDbTableDescriptor.html">RocksDbTableDescriptor</a> or a <a href="javadocs/org/apache/samza/table/descriptors/RemoteTableDescriptor">RemoteTableDescriptor</a>.</p> <p>The following example illustrates how you can use input and output descriptors for a Kafka system, and a table descriptor for a local RocksDB table within your application:</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> - <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStreamApplication</span> <span class="kd">implements</span> <span class="n">StreamApplication</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> + <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStreamApplication</span> <span class="kd">implements</span> <span class="nc">StreamApplication</span> <span class="o">{</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="n">StreamApplicationDescriptor</span> <span class="n">appDescriptor</span><span class="o">)</span> <span class="o">{</span> - <span class="n">KafkaSystemDescriptor</span> <span class="n">ksd</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KafkaSystemDescriptor</span><span class="o">(</span><span class="s">"kafka"</span><span class="o">)</span> - <span class="o">.</span><span class="na">withConsumerZkConnect</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"..."</span><span class="o">))</span> - <span class="o">.</span><span class="na">withProducerBootstrapServers</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"..."</span><span class="o">,</span> <span class="s">"..."</span><span class="o">));</span> - <span class="n">KafkaInputDescriptor</span><span class="o"><</span><span class="n">PageView</span><span class="o">></span> <span class="n">kid</span> <span class="o">=</span> - <span class="n">ksd</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">KafkaOutputDescriptor</span><span class="o"><</span><span class="n">DecoratedPageView</span><span class="o">></span> <span class="n">kod</span> <span class="o">=</span> - <span class="n">ksd</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="nc">StreamApplicationDescriptor</span> <span class="n">appDescriptor</span><span class="o">)</span> <span class="o">{</span> + <span class="nc">KafkaSystemDescriptor</span> <span class="n">ksd</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KafkaSystemDescriptor</span><span class="o">(</span><span class="s">"kafka"</span><span class="o">)</span> + <span class="o">.</span><span class="na">withConsumerZkConnect</span><span class="o">(</span><span class="nc">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"..."</span><span class="o">))</span> + <span class="o">.</span><span class="na">withProducerBootstrapServers</span><span class="o">(</span><span class="nc">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"..."</span><span class="o">,</span> <span class="s">"..."</span><span class="o">));</span> + <span class="nc">KafkaInputDescriptor</span><span class="o"><</span><span class="nc">PageView</span><span class="o">></span> <span class="n">kid</span> <span class="o">=</span> + <span class="n">ksd</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="nc">KafkaOutputDescriptor</span><span class="o"><</span><span class="nc">DecoratedPageView</span><span class="o">></span> <span class="n">kod</span> <span class="o">=</span> + <span class="n">ksd</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">RocksDbTableDescriptor</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">td</span> <span class="o">=</span> - <span class="k">new</span> <span class="n">RocksDbTableDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">viewCounts</span><span class="err">â</span><span class="o">,</span> <span class="n">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="n">IntegerSerde</span><span class="o">()));</span> + <span class="nc">RocksDbTableDescriptor</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="n">td</span> <span class="o">=</span> + <span class="k">new</span> <span class="nf">RocksDbTableDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">viewCounts</span><span class="err">â</span><span class="o">,</span> <span class="nc">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="nc">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="nc">IntegerSerde</span><span class="o">()));</span> <span class="c1">// Implement your processing logic here</span> <span class="o">}</span> @@ -721,21 +730,21 @@ can use the <a href="javadocs/org/apache <h4 id="stream-processing-logic">Stream Processing Logic</h4> -<p>Samza provides two sets of APIs to define the main stream processing logic, High Level Streams API and Low Level Task API, via StreamApplication and TaskApplication, respectively. </p> +<p>Samza provides two sets of APIs to define the main stream processing logic, High Level Streams API and Low Level Task API, via StreamApplication and TaskApplication, respectively.</p> <p>High Level Streams API allows you to describe the processing logic in a connected DAG of transformation operators, like the example below:</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BadPageViewFilter</span> <span class="kd">implements</span> <span class="n">StreamApplication</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BadPageViewFilter</span> <span class="kd">implements</span> <span class="nc">StreamApplication</span> <span class="o">{</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="n">StreamApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> - <span class="n">KafkaSystemDescriptor</span> <span class="n">ksd</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KafkaSystemDescriptor</span><span class="o">();</span> - <span class="n">InputDescriptor</span><span class="o"><</span><span class="n">PageView</span><span class="o">></span> <span class="n">pageViewInput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">OutputDescriptor</span><span class="o"><</span><span class="n">DecoratedPageView</span><span class="o">></span> <span class="n">pageViewOutput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">RocksDbTableDescriptor</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">viewCountTable</span> <span class="o">=</span> <span class="k">new</span> <span class="n">RocksDbTableDescriptor</span><span class="o">(</span> - <span class="err">â</span><span class="n">pageViewCountTable</span><span class="err">â</span><span class="o">,</span> <span class="n">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="n">IntegerSerde</span><span class="o">()));</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="nc">StreamApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> + <span class="nc">KafkaSystemDescriptor</span> <span class="n">ksd</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KafkaSystemDescriptor</span><span class="o">();</span> + <span class="nc">InputDescriptor</span><span class="o"><</span><span class="nc">PageView</span><span class="o">></span> <span class="n">pageViewInput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="nc">OutputDescriptor</span><span class="o"><</span><span class="nc">DecoratedPageView</span><span class="o">></span> <span class="n">pageViewOutput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="nc">RocksDbTableDescriptor</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="n">viewCountTable</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RocksDbTableDescriptor</span><span class="o">(</span> + <span class="err">â</span><span class="n">pageViewCountTable</span><span class="err">â</span><span class="o">,</span> <span class="nc">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="nc">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="nc">IntegerSerde</span><span class="o">()));</span> <span class="c1">// Now, implement your main processing logic</span> - <span class="n">MessageStream</span><span class="o"><</span><span class="n">PageView</span><span class="o">></span> <span class="n">pageViews</span> <span class="o">=</span> <span class="n">appDesc</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="n">pageViewInput</span><span class="o">);</span> + <span class="nc">MessageStream</span><span class="o"><</span><span class="nc">PageView</span><span class="o">></span> <span class="n">pageViews</span> <span class="o">=</span> <span class="n">appDesc</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="n">pageViewInput</span><span class="o">);</span> <span class="n">pageViews</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">isValidPageView</span><span class="o">)</span> <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">addProfileInformation</span><span class="o">)</span> <span class="o">.</span><span class="na">sendTo</span><span class="o">(</span><span class="n">pageViewOutput</span><span class="o">);</span> @@ -745,19 +754,19 @@ can use the <a href="javadocs/org/apache <p>Low Level Task API allows you to describe the processing logic in a customized StreamTaskFactory or AsyncStreamTaskFactory, like the example below:</p> -<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BadPageViewFilter</span> <span class="kd">implements</span> <span class="n">TaskApplication</span> <span class="o">{</span> +<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BadPageViewFilter</span> <span class="kd">implements</span> <span class="nc">TaskApplication</span> <span class="o">{</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="n">TaskApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> - <span class="n">KafkaSystemDescriptor</span> <span class="n">kafka</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KafkaSystemDescriptor</span><span class="o">();</span> - <span class="n">InputDescriptor</span><span class="o"><</span><span class="n">PageView</span><span class="o">></span> <span class="n">pageViewInput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">OutputDescriptor</span><span class="o"><</span><span class="n">DecoratedPageView</span><span class="o">></span> <span class="n">pageViewOutput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> - <span class="n">RocksDbTableDescriptor</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">viewCountTable</span> <span class="o">=</span> <span class="k">new</span> <span class="n">RocksDbTableDescriptor</span><span class="o">(</span> - <span class="err">â</span><span class="n">pageViewCountTable</span><span class="err">â</span><span class="o">,</span> <span class="n">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="n">IntegerSerde</span><span class="o">()));</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">describe</span><span class="o">(</span><span class="nc">TaskApplicationDescriptor</span> <span class="n">appDesc</span><span class="o">)</span> <span class="o">{</span> + <span class="nc">KafkaSystemDescriptor</span> <span class="n">kafka</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KafkaSystemDescriptor</span><span class="o">();</span> + <span class="nc">InputDescriptor</span><span class="o"><</span><span class="nc">PageView</span><span class="o">></span> <span class="n">pageViewInput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getInputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">PageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="nc">OutputDescriptor</span><span class="o"><</span><span class="nc">DecoratedPageView</span><span class="o">></span> <span class="n">pageViewOutput</span> <span class="o">=</span> <span class="n">kafka</span><span class="o">.</span><span class="na">getOutputDescriptor</span><span class="o">(</span><span class="err">â</span><span class="n">decorated</span><span class="o">-</span><span class="n">page</span><span class="o">-</span><span class="n">views</span><span class="err">â</span><span class="o">,</span> <span class="k">new</span> <span class="nc">JsonSerdeV2</span><span class="o"><>(</span><span class="nc">DecoratedPageView</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> + <span class="nc">RocksDbTableDescriptor</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="n">viewCountTable</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RocksDbTableDescriptor</span><span class="o">(</span> + <span class="err">â</span><span class="n">pageViewCountTable</span><span class="err">â</span><span class="o">,</span> <span class="nc">KVSerde</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="nc">StringSerde</span><span class="o">(),</span> <span class="k">new</span> <span class="nc">IntegerSerde</span><span class="o">()));</span> <span class="c1">// Now, implement your main processing logic</span> <span class="n">appDesc</span><span class="o">.</span><span class="na">withInputStream</span><span class="o">(</span><span class="n">pageViewInput</span><span class="o">)</span> <span class="o">.</span><span class="na">withOutputStream</span><span class="o">(</span><span class="n">pageViewOutput</span><span class="o">)</span> - <span class="o">.</span><span class="na">withTaskFactory</span><span class="o">(</span><span class="k">new</span> <span class="n">PageViewFilterTaskFactory</span><span class="o">());</span> + <span class="o">.</span><span class="na">withTaskFactory</span><span class="o">(</span><span class="k">new</span> <span class="nc">PageViewFilterTaskFactory</span><span class="o">());</span> <span class="o">}</span> <span class="o">}</span> </code></pre></figure> @@ -766,12 +775,12 @@ can use the <a href="javadocs/org/apache <p>To deploy a Samza application, you need to specify the implementation class for your application and the ApplicationRunner to launch your application. The following is an incomplete example of minimum required configuration to set up the Samza application and the runner. For additional configuration, see the Configuration Reference.</p> -<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span> - <span class="c"># This is the class implementing StreamApplication</span> -<span class="na"> app.class</span><span class="o">=</span><span class="s">com.example.samza.PageViewFilter</span> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"> + # This is the class implementing StreamApplication + app.class=com.example.samza.PageViewFilter - <span class="c"># This is defining the ApplicationRunner class to launch the application</span> -<span class="na"> app.runner.class</span><span class="o">=</span><span class="s">org.apache.samza.runtime.RemoteApplicationRunner</span> + # This is defining the ApplicationRunner class to launch the application + app.runner.class=org.apache.samza.runtime.RemoteApplicationRunner </code></pre></figure>
Modified: samza/site/learn/documentation/latest/api/samza-sql.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/api/samza-sql.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/api/samza-sql.html (original) +++ samza/site/learn/documentation/latest/api/samza-sql.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/api/samza-sql">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/api/samza-sql">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/api/samza-sql">1.6.0</a></li> @@ -640,43 +654,49 @@ --> <h3 id="table-of-contents">Table Of Contents</h3> - <ul> -<li><a href="#introduction">Introduction</a></li> -<li><a href="#code-examples">Code Examples</a></li> -<li><a href="#key-concepts">Key Concepts</a> - -<ul> -<li><a href="#sql-representation">SQL Representation</a></li> -<li><a href="#sql-grammar">SQL Grammar</a></li> -<li><a href="#udfs">UDFs</a></li> -<li><a href="#udf-polymorphism">UDF Polymorphism</a></li> -</ul></li> -<li><a href="#known-limitations">Known Limitations</a></li> + <li><a href="#introduction">Introduction</a></li> + <li><a href="#code-examples">Code Examples</a></li> + <li><a href="#key-concepts">Key Concepts</a> + <ul> + <li><a href="#sql-representation">SQL Representation</a></li> + <li><a href="#sql-grammar">SQL Grammar</a></li> + <li><a href="#udfs">UDFs</a></li> + <li><a href="#udf-polymorphism">UDF Polymorphism</a></li> + </ul> + </li> + <li><a href="#known-limitations">Known Limitations</a></li> </ul> <h3 id="introduction">Introduction</h3> - <p>Samza SQL allows you to define your stream processing logic declaratively as a a SQL query. This allows you to create streaming pipelines without Java code or configuration unless you require -user-defined functions (UDFs). </p> +user-defined functions (UDFs).</p> <h3 id="code-examples">Code Examples</h3> <p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql";>Hello Samza</a> -SQL examples demonstrate how to use Samza SQL API. </p> +SQL examples demonstrate how to use Samza SQL API.</p> <ul> -<li><p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-filter";>filter</a> -demonstrates filtering and insert Samza SQL job.</p></li> -<li><p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-casewhen/src/main/sql";>Case-When</a> -shows how to use <code>CASE WHEN</code> statement, along with <code>UDF</code> to identify qualifying events.</p></li> -<li><p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-stream-table-join";>join</a> + <li> + <p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-filter";>filter</a> +demonstrates filtering and insert Samza SQL job.</p> + </li> + <li> + <p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-casewhen/src/main/sql";>Case-When</a> +shows how to use <code class="language-plaintext highlighter-rouge">CASE WHEN</code> statement, along with <code class="language-plaintext highlighter-rouge">UDF</code> to identify qualifying events.</p> + </li> + <li> + <p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-stream-table-join";>join</a> demonstrates how to peform a stream-table join. Please note that join operation is currently not -fully cooked, and we are actively working on stabilizing it. </p></li> -<li><p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-groupby";>group by</a> -show how to do group by. Similar to Join, Group By is being actively stabilized. </p></li> +fully cooked, and we are actively working on stabilizing it.</p> + </li> + <li> + <p>The <a href="https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/sql/samza-sql-groupby";>group by</a> +show how to do group by. Similar to Join, Group By is being actively stabilized.</p> + </li> </ul> <h3 id="key-concepts">Key Concepts</h3> @@ -688,66 +708,72 @@ Each statement represents a single strea <p>Support for SQL internally uses Apache Calcite, which provides SQL parsing and query planning. The query is automatically translated -to Samza’s high level API and runs on Samza’s execution engine.</p> +to Samzaâs high level API and runs on Samzaâs execution engine.</p> -<p>The mapping from SQL to the Samza’s high level API is a simple -deterministic one-to-one mapping. For example, <code>Select</code>, i.e., -projections, maps to a filter operation, while <code>from</code> maps to -a scan(s) and join(s) - if selecting from multiple streams and tables -- operators, and so on. </p> +<p>The mapping from SQL to the Samzaâs high level API is a simple +deterministic one-to-one mapping. For example, <code class="language-plaintext highlighter-rouge">Select</code>, i.e., +projections, maps to a filter operation, while <code class="language-plaintext highlighter-rouge">from</code> maps to +a scan(s) and join(s) - if selecting from multiple streams and tables</p> +<ul> + <li>operators, and so on.</li> +</ul> <p>The table below lists the supported SQL operations.</p> -<table><thead> -<tr> -<th>Operation</th> -<th>Syntax hints</th> -<th>Comments</th> -</tr> -</thead><tbody> -<tr> -<td>PROJECTION</td> -<td>SELECT/INSERT/UPSERT</td> -<td>See <a href="#sql-grammar">SQL Grammar</a> below</td> -</tr> -<tr> -<td>FILTERING</td> -<td>WHERE expression</td> -<td>See <a href="#sql-grammar">SQL Grammar</a> below</td> -</tr> -<tr> -<td>UDFs</td> -<td>udf_name(args)</td> -<td>In both SELECT and WHERE clause</td> -</tr> -<tr> -<td>JOIN</td> -<td>[LEFT/RIGHT] JOIN .. ON ..</td> -<td>Stream-table inner, left- or right-outer joins. Currently not fully stable.</td> -</tr> -<tr> -<td>AGGREGATION</td> -<td>COUNT ( …) .. GROUP BY</td> -<td>Currently only COUNT is supported, using processing-time based window.</td> -</tr> -</tbody></table> +<table> + <thead> + <tr> + <th>Operation</th> + <th>Syntax hints</th> + <th>Comments</th> + </tr> + </thead> + <tbody> + <tr> + <td>PROJECTION</td> + <td>SELECT/INSERT/UPSERT</td> + <td>See <a href="#sql-grammar">SQL Grammar</a> below</td> + </tr> + <tr> + <td>FILTERING</td> + <td>WHERE expression</td> + <td>See <a href="#sql-grammar">SQL Grammar</a> below</td> + </tr> + <tr> + <td>UDFs</td> + <td>udf_name(args)</td> + <td>In both SELECT and WHERE clause</td> + </tr> + <tr> + <td>JOIN</td> + <td>[LEFT/RIGHT] JOIN .. ON ..</td> + <td>Stream-table inner, left- or right-outer joins. Currently not fully stable.</td> + </tr> + <tr> + <td>AGGREGATION</td> + <td>COUNT ( â¦) .. GROUP BY</td> + <td>Currently only COUNT is supported, using processing-time based window.</td> + </tr> + </tbody> +</table> <h4 id="sql-grammar">SQL Grammar</h4> -<p>Samza SQL’s grammar is a subset of capabilities supported by Calcite’s SQL parser.</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>statement: +<p>Samza SQLâs grammar is a subset of capabilities supported by Calciteâs SQL parser.</p> + +<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>statement: | insert | query - + query: values | { select } - + insert: ( INSERT | UPSERT ) INTO tablePrimary - [ '(' column [, column ]* ')' ] + [ '(' column [, column ]* ')' ] query select: @@ -756,29 +782,31 @@ select: FROM tableExpression [ WHERE booleanExpression ] [ GROUP BY { groupItem [, groupItem ]* } ] - + projectItem: expression [ [ AS ] columnAlias ] | tableAlias . * - + tableExpression: tableReference [, tableReference ]* | tableExpression [ NATURAL ] [ LEFT | RIGHT | FULL ] JOIN tableExpression [ joinCondition ] - + joinCondition: ON booleanExpression - | USING '(' column [, column ]* ')' - + | USING '(' column [, column ]* ')' + tableReference: tablePrimary - [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ] - + [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ] + tablePrimary: [ TABLE ] [ [ catalogName . ] schemaName . ] tableName - + values: VALUES expression [, expression ]* -</code></pre></div> + +</code></pre></div></div> + <h4 id="udfs">UDFs</h4> <p>In addition to existing SQL logical operations, Samza SQL @@ -788,18 +816,18 @@ pipeline corresponding to the SQL.</p> <h4 id="udf-polymorphism">UDF Polymorphism</h4> -<p>Since UDF’s execute method takes an array of generic objects as +<p>Since UDFâs execute method takes an array of generic objects as parameter, Samza SQL UDF framework is flexible enough to support polymorphic udf functions with varying sets of arguments as long as UDF implementations support them.</p> <p>For example in the below sql statement, UDF will be passed an -object array of size 2 with first element containing id of type<br> -“LONG” and second element name of type “String”. The type of the +object array of size 2 with first element containing id of type<br /> +âLONGâ and second element name of type âStringâ. The type of the objects that are passed depends on the type of those fields in Samza SQL message format.</p> -<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span></span><span class="k">select</span> <span class="n">myudf</span><span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">from</span> <span class="k">identity</span><span class="p">.</span><span class="n">profile</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">select</span> <span class="n">myudf</span><span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">from</span> <span class="k">identity</span><span class="p">.</span><span class="n">profile</span></code></pre></figure> <h3 id="known-limitations">Known Limitations</h3> @@ -807,6 +835,7 @@ SQL message format.</p> and projections. We are actively working on supporting stateful operations such as aggregations, windows and joins.</p> + </div> </div>
