Modified: samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html (original) +++ samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -544,30 +550,29 @@ See the License for the specific language governing permissions and limitations under the License. --> - <p>The <a href="https://github.com/apache/samza-hello-samza";>hello-samza</a> project is an example project designed to help you run your first Samza application. It has examples of applications using the Low Level Task API as well as the High Level Streams API.</p> -<p>This tutorial demonstrates a simple wikipedia application created with the High Level Streams API. The <a href="/startup/hello-samza/latest/index.html">Hello Samza tutorial</a> is the Low Level Task API analog to this tutorial. It demonstrates the same logic but is created with the Low Level Task API. The tutorials are designed to be as similar as possible. The primary differences are that with the High Level Streams API we accomplish the equivalent of 3 separate Low Level Task API jobs with a single application, we skip the intermediate topics for simplicity, and we can visualize the execution plan after we start the application.</p> +<p>This tutorial demonstrates a simple wikipedia application created with the High Level Streams API. The [Hello Samza tutorial] (/startup/hello-samza/latest/index.html) is the Low Level Task API analog to this tutorial. It demonstrates the same logic but is created with the Low Level Task API. The tutorials are designed to be as similar as possible. The primary differences are that with the High Level Streams API we accomplish the equivalent of 3 separate Low Level Task API jobs with a single application, we skip the intermediate topics for simplicity, and we can visualize the execution plan after we start the application.</p> <h3 id="get-the-code">Get the Code</h3> <p>Check out the hello-samza project:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git hello-samza -<span class="nb">cd</span> hello-samza +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git hello-samza +<span class="nb">cd </span>hello-samza git checkout latest</code></pre></figure> -<p>This project contains everything you’ll need to run your first Samza application.</p> +<p>This project contains everything youâll need to run your first Samza application.</p> <h3 id="start-a-grid">Start a Grid</h3> -<p>A Samza grid usually comprises three different systems: <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a>, <a href="http://kafka.apache.org/";>Kafka</a>, and <a href="http://zookeeper.apache.org/";>ZooKeeper</a>. The hello-samza project comes with a script called “grid” to help you setup these systems. Start by running:</p> +<p>A Samza grid usually comprises three different systems: <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a>, <a href="http://kafka.apache.org/";>Kafka</a>, and <a href="http://zookeeper.apache.org/";>ZooKeeper</a>. The hello-samza project comes with a script called âgridâ to help you setup these systems. Start by running:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid bootstrap</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/grid bootstrap</code></pre></figure> -<p>This command will download, install, and start ZooKeeper, Kafka, and YARN. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called “deploy” inside hello-samza’s root folder.</p> +<p>This command will download, install, and start ZooKeeper, Kafka, and YARN. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called âdeployâ inside hello-samzaâs root folder.</p> -<p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it to the path where Java is installed on your system.</p> +<p>If you get a complaint that JAVA_HOME is not set, then youâll need to set it to the path where Java is installed on your system.</p> <p>Once the grid command completes, you can verify that YARN is up and running by going to <a href="http://localhost:8088";>http://localhost:8088</a>. This is the YARN UI.</p> @@ -577,65 +582,64 @@ git checkout latest</code></pre></figure <p>NOTE: if you are building from the latest branch of hello-samza project, make sure that you run the following step from your local Samza project first:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew publishToMavenLocal</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./gradlew publishToMavenLocal</code></pre></figure> <p>Then, you can continue w/ the following command in hello-samza project:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -mkdir -p deploy/samza -tar -xvf ./target/hello-samza-1.7.0-SNAPSHOT-dist.tar.gz -C deploy/samza</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn clean package +<span class="nb">mkdir</span> <span class="nt">-p</span> deploy/samza +<span class="nb">tar</span> <span class="nt">-xvf</span> ./target/hello-samza-1.9.0-SNAPSHOT-dist.tar.gz <span class="nt">-C</span> deploy/samza</code></pre></figure> <h3 id="run-a-samza-application">Run a Samza Application</h3> -<p>After you’ve built your Samza package, you can start the app on the grid using the run-app.sh script.</p> +<p>After youâve built your Samza package, you can start the app on the grid using the run-app.sh script.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties</code></pre></figure> <p>The app will do all of the following:</p> <ol> -<li>Consume 3 feeds of real-time edits from Wikipedia</li> -<li>Parse the events to extract information about the size of the edit, who made the change, etc.</li> -<li>Calculate counts, every ten seconds, for all edits that were made during that window</li> -<li>Output the counts to the wikipedia-stats topic</li> + <li>Consume 3 feeds of real-time edits from Wikipedia</li> + <li>Parse the events to extract information about the size of the edit, who made the change, etc.</li> + <li>Calculate counts, every ten seconds, for all edits that were made during that window</li> + <li>Output the counts to the wikipedia-stats topic</li> </ol> <p>For details about how the app works, take a look at the <a href="hello-samza-high-level-code.html">code walkthrough</a>.</p> <p>Give the job a minute to startup, and then tail the Kafka topic:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./deploy/kafka/bin/kafka-console-consumer.sh <span class="nt">--zookeeper</span> localhost:2181 <span class="nt">--topic</span> wikipedia-stats</code></pre></figure> <p>The messages in the stats topic look like this:</p> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="nl">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure> -<p>Pretty neat, right? Now, check out the YARN UI again (<a href="http://localhost:8088";>http://localhost:8088</a>). This time around, you’ll see your Samza job is running!</p> +<p>Pretty neat, right? Now, check out the YARN UI again (<a href="http://localhost:8088";>http://localhost:8088</a>). This time around, youâll see your Samza job is running!</p> <h3 id="view-the-execution-plan">View the Execution Plan</h3> - <p>Each application goes through an execution planner and you can visualize the execution plan after starting the job by opening the following file in a browser</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/plan.html</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">deploy/samza/bin/plan.html</code></pre></figure> <p>This plan will make more sense after the <a href="hello-samza-high-level-code.html">code walkthrough</a>. For now, just take note that this visualization is available and it is useful for visibility into the structure of the application. For this tutorial, the plan should look something like this:</p> -<p><img src="/img/latest/learn/tutorials/hello-samza-high-level/wikipedia-execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)"/></p> +<p><img src="/img/latest/learn/tutorials/hello-samza-high-level/wikipedia-execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> <h3 id="shutdown">Shutdown</h3> -<p>To shutdown the app, use the same <em>run-app.sh</em> script with an extra <em>–operation=kill</em> argument</p> +<p>To shutdown the app, use the same <em>run-app.sh</em> script with an extra <em>âoperation=kill</em> argument</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties --operation<span class="o">=</span><span class="nb">kill</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties <span class="nt">--operation</span><span class="o">=</span><span class="nb">kill</span></code></pre></figure> -<p>After you’re done, you can clean everything up using the same grid script.</p> +<p>After youâre done, you can clean everything up using the same grid script.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid stop all</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/grid stop all</code></pre></figure> -<p>Congratulations! You’ve now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza application on it. Curious how this application was built? See the <a href="hello-samza-high-level-code.html">code walk-through</a>.</p> +<p>Congratulations! Youâve now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza application on it. Curious how this application was built? See the <a href="hello-samza-high-level-code.html">code walk-through</a>.</p> </div>
Modified: samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html (original) +++ samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -550,35 +556,35 @@ In this tutorial, we will learn how to r <h3 id="get-the-code">Get the Code</h3> -<p>Let’s get started by cloning the hello-samza project</p> +<p>Letâs get started by cloning the hello-samza project</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git hello-samza -<span class="nb">cd</span> hello-samza +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git hello-samza +<span class="nb">cd </span>hello-samza git checkout latest</code></pre></figure> <p>The project comes up with numerous examples and for this tutorial, we will pick the Wikipedia application.</p> <h3 id="setting-up-the-deployment-environment">Setting up the Deployment Environment</h3> -<p>For our Wikipedia application, we require two systems: <a href="http://kafka.apache.org/";>Kafka</a> and <a href="http://zookeeper.apache.org/";>ZooKeeper</a>. The hello-samza project comes with a script called “grid” to help with the environment setup</p> +<p>For our Wikipedia application, we require two systems: <a href="http://kafka.apache.org/";>Kafka</a> and <a href="http://zookeeper.apache.org/";>ZooKeeper</a>. The hello-samza project comes with a script called âgridâ to help with the environment setup</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid standalone</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/grid standalone</code></pre></figure> -<p>This command will download, install, and start ZooKeeper and Kafka. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called “deploy” inside hello-samza’s root folder.</p> +<p>This command will download, install, and start ZooKeeper and Kafka. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called âdeployâ inside hello-samzaâs root folder.</p> -<p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it to the path where Java is installed on your system.</p> +<p>If you get a complaint that JAVA_HOME is not set, then youâll need to set it to the path where Java is installed on your system.</p> <h3 id="building-the-hello-samza-project">Building the Hello Samza Project</h3> <p>NOTE: if you are building from the latest branch of hello-samza project and want to use your local copy of samza, make sure that you run the following step from your local Samza project first</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew publishToMavenLocal</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./gradlew publishToMavenLocal</code></pre></figure> <p>With the environment setup complete, let us move on to building the hello-samza project. Execute the following commands:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -mkdir -p deploy/samza -tar -xvf ./target/hello-samza-1.7.0-SNAPSHOT-dist.tar.gz -C deploy/samza</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn clean package +<span class="nb">mkdir</span> <span class="nt">-p</span> deploy/samza +<span class="nb">tar</span> <span class="nt">-xvf</span> ./target/hello-samza-1.9.0-SNAPSHOT-dist.tar.gz <span class="nt">-C</span> deploy/samza</code></pre></figure> <p>We are now all set to deploy the application locally.</p> @@ -586,7 +592,7 @@ tar -xvf ./target/hello-samza-1.7.0-SNAP <p>In order to run the application, we will use the <em>run-wikipedia-zk-application</em> script.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-wikipedia-zk-application.sh</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./deploy/samza/bin/run-wikipedia-zk-application.sh</code></pre></figure> <p>The above command executes the helper script which invokes the <em>WikipediaZkLocalApplication</em> main class with the appropriate job configurations as command line arguments. The main class is an application wrapper that initializes the application and passes it to the local runner for execution. It is blocking and waits for the <em>LocalApplicationRunner</em> to finish.</p> @@ -594,30 +600,30 @@ that initializes the application and pas <p>To run your own application using ZooKeeper deployment model, you would need something similar to <em>WikipediaZkLocalApplication</em> class that initializes your application and uses the <em>LocalApplicationRunner</em> to run it. To learn more about the internals checkout <a href="/startup/preview/">deployment-models</a> documentation and the <a href="/learn/documentation/latest/jobs/configuration-table.html">configurations</a> table.</p> -<p>Getting back to our example, the application consumes a feed of real-time edits from Wikipedia, and produces them to a Kafka topic called “wikipedia-stats”. Give the job a minute to startup, and then tail the Kafka topic. To do so, run the following command:</p> +<p>Getting back to our example, the application consumes a feed of real-time edits from Wikipedia, and produces them to a Kafka topic called âwikipedia-statsâ. Give the job a minute to startup, and then tail the Kafka topic. To do so, run the following command:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./deploy/kafka/bin/kafka-console-consumer.sh <span class="nt">--zookeeper</span> localhost:2181 <span class="nt">--topic</span> wikipedia-stats</code></pre></figure> <p>The messages in the stats topic should look like the sample below:</p> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="nl">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span><span class="w"> +</span><span class="p">{</span><span class="nl">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nl">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nl">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nl">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nl">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure> <p>Excellent! Now that the job is running, open the <em>plan.html</em> file under <em>deploy/samza/bin</em> directory to take a look at the execution plan for the Wikipedia application. The execution plan is a colorful graphic representing various stages of your application and how they are connected. Here is a sample plan visualization:</p> -<p><img src="/img/latest/learn/tutorials/hello-samza-high-level/wikipedia-execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)"/></p> +<p><img src="/img/latest/learn/tutorials/hello-samza-high-level/wikipedia-execution-plan.png" alt="Execution plan" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /></p> <h3 id="shutdown">Shutdown</h3> <p>The Wikipedia application can be shutdown by terminating the <em>run-wikipedia-zk-application</em> script. We can use the <em>grid</em> script to tear down the local environment (<a href="http://kafka.apache.org/";>Kafka</a> and <a href="http://zookeeper.apache.org/";>Zookeeper</a>).</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/grid stop all</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bin/grid stop all</code></pre></figure> -<p>Congratulations! You’ve now successfully run a Samza application using ZooKeeper deployment model. Next up, check out the <a href="/startup/preview/">deployment-models</a> and <a href="/startup/preview.html">high level API</a> pages.</p> +<p>Congratulations! Youâve now successfully run a Samza application using ZooKeeper deployment model. Next up, check out the <a href="/startup/preview/">deployment-models</a> and <a href="/startup/preview.html">high level API</a> pages.</p> </div> Modified: samza/site/learn/tutorials/latest/index.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/index.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/index.html (original) +++ samza/site/learn/tutorials/latest/index.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -544,7 +550,6 @@ See the License for the specific language governing permissions and limitations under the License. --> - <!-- Uncomment after these features are fully released [[Preview] Hello Samza High Level API Zookeeper Deployment](hello-samza-high-level-zk.html) @@ -581,7 +586,7 @@ <a href="group-by-count.html">Group-by and Counting</a><br/> <a href="initialize-close.html">Initializing and Closing</a><br/> <a href="windowing.html">Windowing</a><br/> -<a href="committing.html">Committing</a><br/> --> +<a href="committing.html">Committing</a><br/> \--> </div> Modified: samza/site/learn/tutorials/latest/remote-debugging-samza.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/remote-debugging-samza.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/remote-debugging-samza.html (original) +++ samza/site/learn/tutorials/latest/remote-debugging-samza.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -545,69 +551,69 @@ limitations under the License. --> -<p>Let’s use Eclipse to attach a remote debugger to a Samza container. If you’re an IntelliJ user, you’ll have to fill in the blanks, but the process should be pretty similar. This tutorial assumes you’ve already run through the <a href="../../../startup/hello-samza/latest/">Hello Samza</a> tutorial.</p> +<p>Letâs use Eclipse to attach a remote debugger to a Samza container. If youâre an IntelliJ user, youâll have to fill in the blanks, but the process should be pretty similar. This tutorial assumes youâve already run through the <a href="../../../startup/hello-samza/latest/">Hello Samza</a> tutorial.</p> <h3 id="get-the-code">Get the Code</h3> <p>Start by checking out Samza, so we have access to the source.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone http://git-wip-us.apache.org/repos/asf/samza.git</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone http://git-wip-us.apache.org/repos/asf/samza.git</code></pre></figure> <p>Next, grab hello-samza.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://gitbox.apache.org/repos/asf/samza-hello-samza.git</code></pre></figure> <h3 id="setup-the-environment">Setup the Environment</h3> -<p>Now, let’s setup the Eclipse project files.</p> +<p>Now, letâs setup the Eclipse project files.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> samza +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd </span>samza ./gradlew eclipse</code></pre></figure> -<p>Let’s also release Samza to Maven’s local repository, so hello-samza has access to the JARs that it needs.</p> +<p>Letâs also release Samza to Mavenâs local repository, so hello-samza has access to the JARs that it needs.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew -PscalaSuffix<span class="o">=</span><span class="m">2</span>.11 clean publishToMavenLocal</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./gradlew <span class="nt">-PscalaSuffix</span><span class="o">=</span>2.11 clean publishToMavenLocal</code></pre></figure> -<p>Next, open Eclipse, and import the Samza source code into your workspace: “File” > “Import” > “Existing Projects into Workspace” > “Browse”. Select ‘samza’ folder, and hit ‘finish’.</p> +<p>Next, open Eclipse, and import the Samza source code into your workspace: âFileâ > âImportâ > âExisting Projects into Workspaceâ > âBrowseâ. Select âsamzaâ folder, and hit âfinishâ.</p> <h3 id="enable-remote-debugging">Enable Remote Debugging</h3> <p>Now, go back to the hello-samza project, and edit ./src/main/config/wikipedia-feed.properties to add the following line:</p> -<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">task.opts</span><span class="o">=</span><span class="s">-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties">task.opts=-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y</code></pre></figure> -<p>The <a href="../../documentation/latest/jobs/configuration-table.html">task.opts</a> configuration parameter is a way to override Java parameters at runtime for your Samza containers. In this example, we’re setting the agentlib parameter to enable remote debugging on localhost, port 9009. In a more realistic environment, you might also set Java heap settings (-Xmx, -Xms, etc), as well as garbage collection and logging settings.</p> +<p>The <a href="../../documentation/latest/jobs/configuration-table.html">task.opts</a> configuration parameter is a way to override Java parameters at runtime for your Samza containers. In this example, weâre setting the agentlib parameter to enable remote debugging on localhost, port 9009. In a more realistic environment, you might also set Java heap settings (-Xmx, -Xms, etc), as well as garbage collection and logging settings.</p> -<p><em>NOTE: If you’re running multiple Samza containers on the same machine, there is a potential for port collisions. You must configure your task.opts to assign different ports for different Samza jobs. If a Samza job has more than one container (e.g. if you’re using YARN with job.container.count=2), those containers must be run on different machines.</em></p> +<p><em>NOTE: If youâre running multiple Samza containers on the same machine, there is a potential for port collisions. You must configure your task.opts to assign different ports for different Samza jobs. If a Samza job has more than one container (e.g. if youâre using YARN with job.container.count=2), those containers must be run on different machines.</em></p> <h3 id="start-the-grid">Start the Grid</h3> -<p>Now that the Samza job has been setup to enable remote debugging when a Samza container starts, let’s start the ZooKeeper, Kafka, and YARN.</p> +<p>Now that the Samza job has been setup to enable remote debugging when a Samza container starts, letâs start the ZooKeeper, Kafka, and YARN.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/grid</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bin/grid</code></pre></figure> -<p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it. This can be done on OSX by running:</p> +<p>If you get a complaint that JAVA_HOME is not set, then youâll need to set it. This can be done on OSX by running:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">JAVA_HOME</span><span class="o">=</span><span class="k">$(</span>/usr/libexec/java_home<span class="k">)</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="si">$(</span>/usr/libexec/java_home<span class="si">)</span></code></pre></figure> <p>Once the grid starts, you can start the wikipedia-feed Samza job.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -mkdir -p deploy/samza -tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C deploy/samza -deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn clean package +<span class="nb">mkdir</span> <span class="nt">-p</span> deploy/samza +<span class="nb">tar</span> <span class="nt">-xvf</span> ./target/hello-samza-1.1.0-dist.tar.gz <span class="nt">-C</span> deploy/samza +deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> -<p>When the wikipedia-feed job starts up, a single Samza container will be created to process all incoming messages. This is the container that we’ll want to connect to from the remote debugger.</p> +<p>When the wikipedia-feed job starts up, a single Samza container will be created to process all incoming messages. This is the container that weâll want to connect to from the remote debugger.</p> <h3 id="connect-the-remote-debugger">Connect the Remote Debugger</h3> -<p>Switch back to Eclipse, and set a break point in TaskInstance.process by clicking on a line inside TaskInstance.process, and clicking “Run” > “Toggle Breakpoint”. A blue circle should appear to the left of the line. This will let you see incoming messages as they arrive.</p> +<p>Switch back to Eclipse, and set a break point in TaskInstance.process by clicking on a line inside TaskInstance.process, and clicking âRunâ > âToggle Breakpointâ. A blue circle should appear to the left of the line. This will let you see incoming messages as they arrive.</p> -<p>Setup a remote debugging session: “Run” > “Debug Configurations…” > right click on “Remote Java Application” > “New”. Set the name to ‘wikipedia-feed-debug’. Set the port to 9009 (matching the port in the task.opts configuration). Click “Source” > “Add…” > “Java Project”. Select all of the Samza projects that you imported (i.e. samza-api, samza-core, etc). If you would like to set breakpoints in your own Stream task, also add the project that contains your StreamTask implementation. Click ‘Debug’.</p> +<p>Setup a remote debugging session: âRunâ > âDebug Configurationsâ¦â > right click on âRemote Java Applicationâ > âNewâ. Set the name to âwikipedia-feed-debugâ. Set the port to 9009 (matching the port in the task.opts configuration). Click âSourceâ > âAddâ¦â > âJava Projectâ. Select all of the Samza projects that you imported (i.e. samza-api, samza-core, etc). If you would like to set breakpoints in your own Stream task, also add the project that contains your StreamTask implementation. Click âDebugâ.</p> -<p>After a few moments, Eclipse should connect to the wikipedia-feed job, and ask you to switch to Debug mode. Once in debug, you’ll see that it’s broken at the TaskInstance.process method. From here, you can step through code, inspect variable values, etc.</p> +<p>After a few moments, Eclipse should connect to the wikipedia-feed job, and ask you to switch to Debug mode. Once in debug, youâll see that itâs broken at the TaskInstance.process method. From here, you can step through code, inspect variable values, etc.</p> -<p>Congratulations, you’ve got a remote debug connection to your StreamTask!</p> +<p>Congratulations, youâve got a remote debug connection to your StreamTask!</p> </div> Modified: samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html (original) +++ samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -545,38 +551,40 @@ limitations under the License. --> -<p>This tutorial is to help you run <a href="../../../startup/hello-samza/latest/">Hello Samza</a> if you can not connect to the internet. </p> +<p>This tutorial is to help you run <a href="../../../startup/hello-samza/latest/">Hello Samza</a> if you can not connect to the internet.</p> <h3 id="test-your-connection">Test Your Connection</h3> <p>Ping irc.wikimedia.org. Sometimes the firewall in your company blocks this service.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>telnet irc.wikimedia.org <span class="m">6667</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">telnet irc.wikimedia.org 6667</code></pre></figure> <p>You should see something like this:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Trying 208.80.152.178... + +<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Trying 208.80.152.178... Connected to ekrem.wikimedia.org. -Escape character is '^]'. +Escape character is '^]'. NOTICE AUTH :*** Processing connection to irc.pmtpa.wikimedia.org NOTICE AUTH :*** Looking up your hostname... NOTICE AUTH :*** Checking Ident NOTICE AUTH :*** Found your hostname -</code></pre></div> +</code></pre></div></div> + <p>Otherwise, you may have the connection problem.</p> <h3 id="use-local-data-to-run-hello-samza">Use Local Data to Run Hello Samza</h3> <p>We provide an alternative to get wikipedia feed data. Instead of running</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> <p>You will run</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/produce-wikipedia-raw-data.sh</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bin/produce-wikipedia-raw-data.sh</code></pre></figure> <p>This script will read wikipedia feed data from local file and produce them to the Kafka broker. By default, it produces to localhost:9092 as the Kafka broker and uses localhost:2181 as zookeeper. You can overwrite them:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/produce-wikipedia-raw-data.sh -b yourKafkaBrokerAddress -z yourZookeeperAddress</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bin/produce-wikipedia-raw-data.sh <span class="nt">-b</span> yourKafkaBrokerAddress <span class="nt">-z</span> yourZookeeperAddress</code></pre></figure> <p>Now you can go back to Generate Wikipedia Statistics section in <a href="../../../startup/hello-samza/latest/">Hello Samza</a> and follow the remaining steps.</p> @@ -584,10 +592,11 @@ NOTICE AUTH :*** Found your hostname <p>The goal of</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> <p>is to deploy a Samza job which listens to wikipedia API, receives the feed in realtime and produces the feed to the Kafka topic wikipedia-raw. The alternative in this tutorial is reading local wikipedia feed in an infinite loop and producing the data to Kafka wikipedia-raw. The follow-up job, wikipedia-parser is getting data from Kafka topic wikipedia-raw, so as long as we have correct data in Kafka topic wikipedia-raw, we are fine. All Samza jobs are connected by the Kafka and do not depend on each other.</p> + </div> </div> Modified: samza/site/learn/tutorials/latest/run-in-multi-node-yarn.html URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/run-in-multi-node-yarn.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/tutorials/latest/run-in-multi-node-yarn.html (original) +++ samza/site/learn/tutorials/latest/run-in-multi-node-yarn.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -545,7 +551,7 @@ limitations under the License. --> -<p>You must successfully run the <a href="../../../startup/hello-samza/latest/">hello-samza</a> project in a single-node YARN by following the <a href="../../../startup/hello-samza/latest/">hello-samza</a> tutorial. Now it’s time to run the Samza job in a “real” YARN grid (with more than one node).</p> +<p>You must successfully run the <a href="../../../startup/hello-samza/latest/">hello-samza</a> project in a single-node YARN by following the <a href="../../../startup/hello-samza/latest/">hello-samza</a> tutorial. Now itâs time to run the Samza job in a ârealâ YARN grid (with more than one node).</p> <h2 id="set-up-multi-node-yarn">Set Up Multi-node YARN</h2> @@ -555,57 +561,59 @@ <p>1. Download <a href="http://mirror.symnds.com/software/Apache/hadoop/common/hadoop-2.6.1/hadoop-2.6.1.tar.gz";>YARN 2.6</a> to /tmp and untar it.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /tmp -tar -xvf hadoop-2.6.1.tar.gz -<span class="nb">cd</span> hadoop-2.6.1</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /tmp +<span class="nb">tar</span> <span class="nt">-xvf</span> hadoop-2.6.1.tar.gz +<span class="nb">cd </span>hadoop-2.6.1</code></pre></figure> <p>2. Set up environment variables.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">HADOOP_YARN_HOME</span><span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span> -mkdir conf -<span class="nb">export</span> <span class="nv">HADOOP_CONF_DIR</span><span class="o">=</span><span class="nv">$HADOOP_YARN_HOME</span>/conf</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">HADOOP_YARN_HOME</span><span class="o">=</span><span class="si">$(</span><span class="nb">pwd</span><span class="si">)</span> +<span class="nb">mkdir </span>conf +<span class="nb">export </span><span class="nv">HADOOP_CONF_DIR</span><span class="o">=</span><span class="nv">$HADOOP_YARN_HOME</span>/conf</code></pre></figure> <p>3. Configure YARN setting file.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>cp ./etc/hadoop/yarn-site.xml conf +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cp</span> ./etc/hadoop/yarn-site.xml conf vi conf/yarn-site.xml</code></pre></figure> <p>Add the following property to yarn-site.xml:</p> -<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="nt"><property></span> +<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><property></span> <span class="nt"><name></span>yarn.resourcemanager.hostname<span class="nt"></name></span> <span class="c"><!-- hostname that is accessible from all NMs --></span> <span class="nt"><value></span>yourHostname<span class="nt"></value></span> <span class="nt"></property></span></code></pre></figure> <p>Download and add capacity-schedule.xml.</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>curl http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml?view=co > conf/capacity-scheduler.xml -</code></pre></div> + +<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml?view=co > conf/capacity-scheduler.xml +</code></pre></div></div> + <h3 id="set-up-http-filesystem-for-yarn">Set Up Http Filesystem for YARN</h3> <p>The goal of these steps is to configure YARN to read http filesystem because we will use Http server to deploy Samza job package. If you want to use HDFS to deploy Samza job package, you can skip step 4~6 and follow <a href="deploy-samza-job-from-hdfs.html">Deploying a Samza Job from HDFS</a></p> <p>4. Download Scala package and untar it.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /tmp -curl http://www.scala-lang.org/files/archive/scala-2.11.8.tgz > scala-2.11.8.tgz -tar -xvf scala-2.11.8.tgz</code></pre></figure> - -<p>5. Add Scala, its log jars, and Samza’s HttpFileSystem implementation.</p> - -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>cp /tmp/scala-2.11.8/lib/scala-compiler.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib -cp /tmp/scala-2.11.8/lib/scala-library.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib -curl -L http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/clapper/grizzled-slf4j_2.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar > <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar -curl -L http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/apache/samza/samza-yarn_2.11/0.12.0/samza-yarn_2.11-0.12.0.jar > <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/samza-yarn_2.11-0.12.0.jar -curl -L http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/apache/samza/samza-core_2.11/0.12.0/samza-core_2.11-0.12.0.jar > <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/samza-core_2.11-0.12.0.jar</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /tmp +curl http://www.scala-lang.org/files/archive/scala-2.11.8.tgz <span class="o">></span> scala-2.11.8.tgz +<span class="nb">tar</span> <span class="nt">-xvf</span> scala-2.11.8.tgz</code></pre></figure> + +<p>5. Add Scala, its log jars, and Samzaâs HttpFileSystem implementation.</p> + +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cp</span> /tmp/scala-2.11.8/lib/scala-compiler.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib +<span class="nb">cp</span> /tmp/scala-2.11.8/lib/scala-library.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib +curl <span class="nt">-L</span> http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/clapper/grizzled-slf4j_2.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar <span class="o">></span> <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar +curl <span class="nt">-L</span> http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/apache/samza/samza-yarn_2.11/0.12.0/samza-yarn_2.11-0.12.0.jar <span class="o">></span> <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/samza-yarn_2.11-0.12.0.jar +curl <span class="nt">-L</span> http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/apache/samza/samza-core_2.11/0.12.0/samza-core_2.11-0.12.0.jar <span class="o">></span> <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/samza-core_2.11-0.12.0.jar</code></pre></figure> <p>6. Add http configuration in core-site.xml (create the core-site.xml file and add content).</p> -<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span>vi $HADOOP_YARN_HOME/conf/core-site.xml</code></pre></figure> +<figure class="highlight"><pre><code class="language-xml" data-lang="xml">vi $HADOOP_YARN_HOME/conf/core-site.xml</code></pre></figure> <p>Add the following code:</p> -<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span> +<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span> <span class="nt"><configuration></span> <span class="nt"><property></span> <span class="nt"><name></span>fs.http.impl<span class="nt"></name></span> @@ -617,12 +625,12 @@ curl -L http://search.maven.org/remoteco <p>7. Basically, you copy the hadoop file in your host machine to slave machines. (172.21.100.35, in my case):</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>scp -r . <span class="m">172</span>.21.100.35:/tmp/hadoop-2.6.1 -<span class="nb">echo</span> <span class="m">172</span>.21.100.35 > conf/slaves +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">scp <span class="nt">-r</span> <span class="nb">.</span> 172.21.100.35:/tmp/hadoop-2.6.1 +<span class="nb">echo </span>172.21.100.35 <span class="o">></span> conf/slaves sbin/start-yarn.sh</code></pre></figure> <ul> -<li>If you get “172.21.100.35: Error: JAVA<em>HOME is not set and could not be found.”, you’ll need to add a conf/hadoop-env.sh file to the machine with the failure (172.21.100.35, in this case), which has “export JAVA</em>HOME=/export/apps/jdk/JDK-1<em>8</em>0<em>45” (or wherever your JAVA</em>HOME actually is).</li> + <li>If you get â172.21.100.35: Error: JAVA_HOME is not set and could not be found.â, youâll need to add a conf/hadoop-env.sh file to the machine with the failure (172.21.100.35, in this case), which has âexport JAVA_HOME=/export/apps/jdk/JDK-1_8_0_45â (or wherever your JAVA_HOME actually is).</li> </ul> <p>8. Validate that your nodes are up by visiting http://yourHostname:8088/cluster/nodes.</p> @@ -633,41 +641,42 @@ sbin/start-yarn.sh</code></pre></figure> <p>1. Download Samza and publish it to Maven local repository.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /tmp +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /tmp git clone http://git-wip-us.apache.org/repos/asf/samza.git -<span class="nb">cd</span> samza +<span class="nb">cd </span>samza ./gradlew clean publishToMavenLocal <span class="nb">cd</span> ..</code></pre></figure> <p>2. Download hello-samza project and change the job properties file.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone git://github.com/linkedin/hello-samza.git -<span class="nb">cd</span> hello-samza +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone git://github.com/linkedin/hello-samza.git +<span class="nb">cd </span>hello-samza vi src/main/config/wikipedia-feed.properties</code></pre></figure> <p>Change the yarn.package.path property to be:</p> -<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">yarn.package.path</span><span class="o">=</span><span class="s">http://yourHostname:8000/target/hello-samza-1.1.0-dist.tar.gz</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties">yarn.package.path=http://yourHostname:8000/target/hello-samza-1.1.0-dist.tar.gz</code></pre></figure> <p>3. Compile hello-samza.</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -mkdir -p deploy/samza -tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C deploy/samza</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">mvn clean package +<span class="nb">mkdir</span> <span class="nt">-p</span> deploy/samza +<span class="nb">tar</span> <span class="nt">-xvf</span> ./target/hello-samza-1.1.0-dist.tar.gz <span class="nt">-C</span> deploy/samza</code></pre></figure> <p>4. Deploy Samza job package to Http server..</p> <p>Open a new terminal, and run:</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /tmp/hello-samza <span class="o">&&</span> python -m SimpleHTTPServer</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /tmp/hello-samza <span class="o">&&</span> python <span class="nt">-m</span> SimpleHTTPServer</code></pre></figure> <p>Go back to the original terminal (not the one running the HTTP server):</p> -<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/run-app.sh --config-path<span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> +<figure class="highlight"><pre><code class="language-bash" data-lang="bash">deploy/samza/bin/run-app.sh <span class="nt">--config-path</span><span class="o">=</span><span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure> + +<p>Go to http://yourHostname:8088 and find the wikipedia-feed job. Click on the ApplicationMaster link to see that itâs running.</p> -<p>Go to http://yourHostname:8088 and find the wikipedia-feed job. Click on the ApplicationMaster link to see that it’s running.</p> +<p>Congratulations! You now run the Samza job in a ârealâ YARN grid!</p> -<p>Congratulations! You now run the Samza job in a “real” YARN grid!</p> </div>
