Modified: samza/site/learn/documentation/latest/rest/resources/jobs.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/rest/resources/jobs.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/rest/resources/jobs.html (original) +++ samza/site/learn/documentation/latest/rest/resources/jobs.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/rest/resources/jobs">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/rest/resources/jobs">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/rest/resources/jobs">1.6.0</a></li> @@ -641,74 +655,70 @@ <p>Exposes operations at the job scope (as opposed to the cluster, container, or task scope). The initial implementation includes the ability to list all jobs, get the status of a particular job, and start or stop an individual job.</p> -<h1 id="api">API</h1> - -<p>The following sections provide general information about the response structure and detailed descriptions of each of the requests.</p> +<p>#API +The following sections provide general information about the response structure and detailed descriptions of each of the requests.</p> <h2 id="response-structure">Response Structure</h2> - <p>All responses will contain either a job status or an error message.</p> <p><strong>Job Status</strong></p> <p>Job status will be of the form:</p> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-parser"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span><span class="s2">"1"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-parser"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="s2">"1"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> -<p><code>status</code> is the abstract Samza status for the job. Initially it will be one of {STARTING, STARTED, STOPPED, UNKNOWN}.</p> +<p><code class="language-plaintext highlighter-rouge">status</code> is the abstract Samza status for the job. Initially it will be one of {STARTING, STARTED, STOPPED, UNKNOWN}.</p> -<p><code>statusDetail</code> is the implementation-specific status for the job. For YARN, it will be one of the values in the YarnApplicationState enum.</p> +<p><code class="language-plaintext highlighter-rouge">statusDetail</code> is the implementation-specific status for the job. For YARN, it will be one of the values in the YarnApplicationState enum.</p> <p><strong>Error Message</strong></p> <p>Every error response have the following structure:</p> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Unrecognized status parameter: null"</span> -<span class="p">}</span></code></pre></figure> - -<p><code>message</code> is the only field in the response and contains a description of the problem. -<br/></p> - -<h2 id="get-all-jobs">Get All Jobs</h2> - -<p>Lists all the jobs installed on the host and provides their status.</p> - -<h6 id="request">Request</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>GET /v1/jobs -</code></pre></div> -<h6 id="response">Response</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Status: 200 OK -</code></pre></div> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">[</span> - - <span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-parser"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span><span class="s2">"1"</span> - <span class="p">},</span> - <span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STARTED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"RUNNING"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-feed"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span><span class="s2">"1"</span> - <span class="p">},</span> - <span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="kc">null</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-stats"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span><span class="s2">"1"</span> - <span class="p">}</span> -<span class="p">]</span></code></pre></figure> - -<h6 id="response-codes">Response codes</h6> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unrecognized status parameter: null"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> + +<p><code class="language-plaintext highlighter-rouge">message</code> is the only field in the response and contains a description of the problem. +<br /></p> + +<p>##Get All Jobs +Lists all the jobs installed on the host and provides their status.</p> + +<p>######Request + GET /v1/jobs</p> + +<p>######Response + Status: 200 OK</p> + +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">[</span><span class="w"> + + </span><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-parser"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="s2">"1"</span><span class="w"> + </span><span class="p">},</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STARTED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"RUNNING"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-feed"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="s2">"1"</span><span class="w"> + </span><span class="p">},</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="kc">null</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-stats"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="s2">"1"</span><span class="w"> + </span><span class="p">}</span><span class="w"> +</span><span class="p">]</span></code></pre></figure> +<p>######Response codes</p> <table class="table table-condensed table-bordered table-striped"> <thead> <tr> @@ -722,40 +732,36 @@ </tr> <tr> <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Timeout waiting for job status."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Timeout waiting for job status."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> </tbody> </table> +<p><br /></p> -<p><br/></p> +<p>##Get Job +Gets the status of the specified job.</p> -<h2 id="get-job">Get Job</h2> +<p>######Format + GET /v1/jobs/{jobName}/{jobId} +The <code class="language-plaintext highlighter-rouge">{jobName}</code> and <code class="language-plaintext highlighter-rouge">{jobId}</code> path parameters reflect the values of âjob.nameâ and âjob.idâ in the job config.</p> -<p>Gets the status of the specified job.</p> +<p>######Request + GET /v1/jobs/wikipedia-feed/1</p> -<h6 id="format">Format</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>GET /v1/jobs/{jobName}/{jobId} -</code></pre></div> -<p>The <code>{jobName}</code> and <code>{jobId}</code> path parameters reflect the values of ‘job.name’ and ‘job.id’ in the job config.</p> - -<h6 id="request">Request</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>GET /v1/jobs/wikipedia-feed/1 -</code></pre></div> -<h6 id="response">Response</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Status: 200 OK -</code></pre></div> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STARTED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"RUNNING"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-feed"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span><span class="s2">"1"</span> -<span class="p">}</span></code></pre></figure> +<p>######Response + Status: 200 OK</p> -<h6 id="response-codes">Response codes</h6> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STARTED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"RUNNING"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="s2">"wikipedia-feed"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="s2">"1"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> +<p>######Response codes</p> <table class="table table-condensed table-bordered table-striped"> <thead> <tr> @@ -769,46 +775,43 @@ </tr> <tr> <td>404 Not Found</td><td>The specified job+instance was not found. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Job wikipedia-¯</span><span class="se">\_</span><span class="s2">(ã)_/¯-feed instance 1 is not installed on this host."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> <tr> <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Timeout waiting for job status."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Timeout waiting for job status."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> </tbody> </table> +<p><br /></p> + +<p>##Start Job +Starts the job with the specified app name if itâs not already started. The command will return when it has initiated the start operation.</p> -<p><br/></p> +<p>######Format + PUT /v1/jobs/{jobName}/{jobId}?status=started</p> -<h2 id="start-job">Start Job</h2> +<p>Form parameter <code class="language-plaintext highlighter-rouge">status</code> is the intended status of the job at the end of the request.</p> -<p>Starts the job with the specified app name if it’s not already started. The command will return when it has initiated the start operation.</p> +<p>######Example + PUT /v1/jobs/wikipedia-feed/1?status=started +######Response + Status: 202 Accepted</p> -<h6 id="format">Format</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>PUT /v1/jobs/{jobName}/{jobId}?status=started -</code></pre></div> -<p>Form parameter <code>status</code> is the intended status of the job at the end of the request.</p> - -<h6 id="example">Example</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>PUT /v1/jobs/wikipedia-feed/1?status=started -</code></pre></div> -<h6 id="response">Response</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Status: 202 Accepted -</code></pre></div> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STARTING"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"ACCEPTED"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span> <span class="s2">"wikipedia-feed"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span> <span class="s2">"1"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STARTING"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"ACCEPTED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikipedia-feed"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> -<h6 id="response-codes">Response codes</h6> +<p>######Response codes</p> <table class="table table-condensed table-bordered table-striped"> <thead> @@ -823,53 +826,50 @@ </tr> <tr> <td>400 Bad Request</td><td>There was a problem with the request. e.g. an invalid status parameter. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Unrecognized status parameter: null"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unrecognized status parameter: null"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> <tr> <td>404 Not Found</td><td>The specified job+instance was not found. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Job wikipedia-¯</span><span class="se">\_</span><span class="s2">(ã)_/¯-feed instance 1 is not installed on this host."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> <tr> <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Timeout waiting for job status."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Timeout waiting for job status."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> - </tbody> -</table> + </tbody> +</table> +<br /> + +##Stop Job +Stops the job with the specified app name if it's not already stopped. -<p><br/></p> +######Format + PUT /v1/jobs/{jobName}/{jobId}?status=stopped -<h2 id="stop-job">Stop Job</h2> +Form parameter `status` is the intended status of the job at the end of the request. -<p>Stops the job with the specified app name if it’s not already stopped.</p> +######Example + PUT /v1/jobs/wikipedia-feed/1?status=stopped +######Response + Status: 202 Accepted -<h6 id="format">Format</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>PUT /v1/jobs/{jobName}/{jobId}?status=stopped -</code></pre></div> -<p>Form parameter <code>status</code> is the intended status of the job at the end of the request.</p> - -<h6 id="example">Example</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>PUT /v1/jobs/wikipedia-feed/1?status=stopped -</code></pre></div> -<h6 id="response">Response</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Status: 202 Accepted -</code></pre></div> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span> - <span class="nt">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span> - <span class="nt">"jobName"</span><span class="p">:</span> <span class="s2">"wikipedia-feed"</span><span class="p">,</span> - <span class="nt">"jobId"</span><span class="p">:</span> <span class="s2">"1"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"status"</span><span class="p">:</span><span class="s2">"STOPPED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"statusDetail"</span><span class="p">:</span><span class="s2">"KILLED"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikipedia-feed"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"jobId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> -<h6 id="response-codes">Response codes</h6> +######Response codes <table class="table table-condensed table-bordered table-striped"> <thead> <tr> @@ -883,62 +883,56 @@ </tr> <tr> <td>400 Bad Request</td><td>There was a problem with the request. e.g. an invalid status parameter. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Unrecognized status parameter: null"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unrecognized status parameter: null"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> <tr> <td>404 Not Found</td><td>The specified job+instance was not found. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Job wikipedia-¯</span><span class="se">\_</span><span class="s2">(ã)_/¯-feed instance 1 is not installed on this host."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> <tr> <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Timeout waiting for job status."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Timeout waiting for job status."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> - </tbody> -</table> - -<p><br/></p> - -<h1 id="design">Design</h1> - -<h3 id="abstractions">Abstractions</h3> - -<p>There are three primary abstractions used by the JobsResource that users can implement to handle any details specific to their environment.</p> + </tbody> +</table> +<br /> -<ol> -<li> <strong>JobProxy</strong>: The JobProxy is the central point of interacting with Samza jobs. It exposes generic methods to start, stop, and get the status of a Samza job. Implementations of this interface can employ custom code to implement these methods tailored to the specific API for any cluster manager.</li> -<li> <strong>InstallationFinder</strong>: The InstallationFinder provides a generic interface to discover all the installed jobs, hiding any customizations in the job package structure and its location (e.g. local vs remote host). The InstallationFinder also resolves the job configuration, which is used to validate and identify the job.</li> -<li> <strong>JobStatusProvider</strong>: The JobStatusProvider allows the JobProxy to get the status of a job in a generic way. The same interface can be used to get the job status on Yarn, Mesos, or standalone jobs. It also enables different implementations for the same cluster. With Yarn, for example, one implementation may get job status via command line and another via the ResourceManager REST API.</li> -</ol> +# Design +###Abstractions +There are three primary abstractions used by the JobsResource that users can implement to handle any details specific to their environment. -<p>The <a href="jobs.html#configuration">configuration</a> must specify a JobProxy factory class explicitly. By contrast, the InstallationFinder and JobStatusProvider abstractions are natural extensions of the JobProxy and are solely provided to demonstrate a pattern for discovering installed jobs and fetching job status. However, they are not an explicit requirement.</p> +1. **JobProxy**: The JobProxy is the central point of interacting with Samza jobs. It exposes generic methods to start, stop, and get the status of a Samza job. Implementations of this interface can employ custom code to implement these methods tailored to the specific API for any cluster manager. +2. **InstallationFinder**: The InstallationFinder provides a generic interface to discover all the installed jobs, hiding any customizations in the job package structure and its location (e.g. local vs remote host). The InstallationFinder also resolves the job configuration, which is used to validate and identify the job. +3. **JobStatusProvider**: The JobStatusProvider allows the JobProxy to get the status of a job in a generic way. The same interface can be used to get the job status on Yarn, Mesos, or standalone jobs. It also enables different implementations for the same cluster. With Yarn, for example, one implementation may get job status via command line and another via the ResourceManager REST API. -<p>The <code>SimpleYarnJobProxy</code> that ships with Samza REST is intended to demonstrate a functional implementation of a JobProxy which works with the Hello Samza jobs. See the <a href="../../../../tutorials/latest/samza-rest-getting-started.html">tutorial</a> to try it out. You can implement your own JobProxy to adapt the JobsResource to the specifics of your job packaging and deployment model.</p> +The [configuration](jobs.html#configuration) must specify a JobProxy factory class explicitly. By contrast, the InstallationFinder and JobStatusProvider abstractions are natural extensions of the JobProxy and are solely provided to demonstrate a pattern for discovering installed jobs and fetching job status. However, they are not an explicit requirement. -<h3 id="request-flow">Request Flow</h3> +The `SimpleYarnJobProxy` that ships with Samza REST is intended to demonstrate a functional implementation of a JobProxy which works with the Hello Samza jobs. See the [tutorial](../../../../tutorials/latest/samza-rest-getting-started.html) to try it out. You can implement your own JobProxy to adapt the JobsResource to the specifics of your job packaging and deployment model. -<p>After validating each request, the JobsResource invokes the appropriate JobProxy command. The JobProxy uses the InstallationFinder to enumerate the installed jobs and the JobStatusProvider to get the runtime status of the jobs.</p> -<p>The provided <a href="../javadocs/org/apache/samza/rest/proxy/installation/SimpleInstallationFinder.html">SimpleInstallationFinder</a> crawls the file system, starting in the directory specified by the <code>job.installations.path</code> looking for valid Samza job config files. It extracts the <code>job.name</code> and <code>job.id</code> property values and creates an <a href="../javadocs/org/apache/samza/rest/proxy/installation/InstallationRecord.html">InstallationRecord</a> for the each job instance. The InstallationRecord contains all the information needed to start, stop, and get the status for the job.</p> +### Request Flow +After validating each request, the JobsResource invokes the appropriate JobProxy command. The JobProxy uses the InstallationFinder to enumerate the installed jobs and the JobStatusProvider to get the runtime status of the jobs. -<p>The provided <a href="../javadocs/org/apache/samza/rest/proxy/job/YarnRestJobStatusProvider.html">YarnRestJobStatusProvider</a> uses the Resource Manager’s REST API to fetch the job status.</p> +The provided [SimpleInstallationFinder](../javadocs/org/apache/samza/rest/proxy/installation/SimpleInstallationFinder.html) crawls the file system, starting in the directory specified by the `job.installations.path` looking for valid Samza job config files. It extracts the `job.name` and `job.id` property values and creates an [InstallationRecord](../javadocs/org/apache/samza/rest/proxy/installation/InstallationRecord.html) for the each job instance. The InstallationRecord contains all the information needed to start, stop, and get the status for the job. -<p>The <a href="../javadocs/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.html">SimpleYarnJobProxy</a> relies on the scripts in the InstallationRecord scriptFilePath (<code>/bin</code>) directory to start and stop jobs.</p> +The provided [YarnRestJobStatusProvider](../javadocs/org/apache/samza/rest/proxy/job/YarnRestJobStatusProvider.html) uses the Resource Manager's REST API to fetch the job status. -<p>The following is a depiction of the implementation that ships with Samza REST:</p> +The [SimpleYarnJobProxy](../javadocs/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.html) relies on the scripts in the InstallationRecord scriptFilePath (`/bin`) directory to start and stop jobs. -<p><img src="/img/latest/learn/documentation/rest/JobsResource.png" alt="Jobs resource component diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)"/></p> +The following is a depiction of the implementation that ships with Samza REST: -<h2 id="configuration">Configuration</h2> +<img src="/img/latest/learn/documentation/rest/JobsResource.png" alt="Jobs resource component diagram" style="max-width: 100%; height: auto;" onclick="window.open(this.src)" /> -<p>The JobsResource properties should be specified in the same file as the Samza REST configuration. They are specified here for clarity.</p> +## Configuration +The JobsResource properties should be specified in the same file as the Samza REST configuration. They are specified here for clarity. <table class="table table-condensed table-bordered table-striped"> <thead> @@ -955,13 +949,14 @@ <td>job.installations.path</td><td><b>Required:</b> The file system path which contains the Samza job installations. The path must be on the same host as the Samza REST Service. Each installation must be a directory with structure conforming to the expectations of the InstallationRecord implementation used by the JobProxy.</td> </tr> <tr> - <td>job.config.factory.class</td><td>The config factory to use for reading Samza job configs. This is used to fetch the job.name and job.id properties for each job instance in the InstallationRecord. It’s also used to validate that a particular directory within the installations path actually contains Samza jobs. If not specified <pre>org.apache.samza.config.factories.PropertiesConfigFactory</pre> will be used. </td> + <td>job.config.factory.class</td><td>The config factory to use for reading Samza job configs. This is used to fetch the job.name and job.id properties for each job instance in the InstallationRecord. It's also used to validate that a particular directory within the installations path actually contains Samza jobs. If not specified <pre>org.apache.samza.config.factories.PropertiesConfigFactory</pre> will be used. </td> </tr> <tr> <td>yarn.resourcemanager.api.endpoint</td><td> An optional config if YARN is used as the cluster manager. This provides the host and port at which the YARN ResourceManager REST API is exposed. If not specified <pre>localhost:8088</pre> will be used.</td> </tr> </tbody> </table> +</tr></tbody></table></tr></tbody></table> </div>
Modified: samza/site/learn/documentation/latest/rest/resources/tasks.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/rest/resources/tasks.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/rest/resources/tasks.html (original) +++ samza/site/learn/documentation/latest/rest/resources/tasks.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/rest/resources/tasks">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/rest/resources/tasks">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/rest/resources/tasks">1.6.0</a></li> @@ -640,7 +654,7 @@ --> <p>This resource exposes endpoints to support operations at the tasks scope. The initial implementation includes the ability to list all the tasks for a particular job. -This is a sub-resource of the <a href="jobs.html">Jobs Resource »</a> and is not intended to be used independently.</p> +This is a sub-resource of the <a href="jobs.html">Jobs Resource »</a> and is not intended to be used independently.</p> <p>Responses of individual endpoints will vary in accordance with their functionality and scope. However, the error messages of all of the tasks resource end points will be of the following form.</p> @@ -649,39 +663,36 @@ messages of all of the tasks resource en <p>Every error response will have the following structure:</p> -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Unrecognized status parameter: null"</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unrecognized status parameter: null"</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> -<p><code>message</code> is the only field in the response and contains a description of the problem. -<br></p> +<p><code class="language-plaintext highlighter-rouge">message</code> is the only field in the response and contains a description of the problem. +<br /></p> <h2 id="get-all-tasks">Get All Tasks</h2> - <p>Lists the complete details about all the tasks for a particular job</p> -<h6 id="request">Request</h6> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>GET /v1/jobs/{jobName}/{jobId}/tasks -</code></pre></div> -<h6 id="response">Response</h6> - -<p>Status: 200 OK</p> - -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span> <span class="p">[</span> -<span class="p">{</span> - <span class="nt">"preferredHost"</span> <span class="p">:</span> <span class="s2">"samza-preferredHost"</span><span class="p">,</span> - <span class="nt">"taskName"</span> <span class="p">:</span> <span class="s2">"Samza task"</span><span class="p">,</span> - <span class="nt">"containerId"</span> <span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span> - <span class="nt">"partitions"</span> <span class="p">:</span> <span class="p">[{</span> - <span class="nt">"system"</span> <span class="p">:</span> <span class="s2">"kafka"</span><span class="p">,</span> - <span class="nt">"stream"</span> <span class="p">:</span> <span class="s2">"topic-name"</span><span class="p">,</span> - <span class="nt">"partitionId"</span> <span class="p">:</span> <span class="s2">"0"</span> - <span class="p">}]</span> - <span class="p">}</span> - <span class="p">]</span></code></pre></figure> +<p>######Request + GET /v1/jobs/{jobName}/{jobId}/tasks</p> + +<p>######Response +Status: 200 OK</p> -<h6 id="response-codes">Response codes</h6> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="w"> </span><span class="p">[</span><span class="w"> +</span><span class="p">{</span><span class="w"> + </span><span class="nl">"preferredHost"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"samza-preferredHost"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"taskName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Samza task"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"containerId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"partitions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w"> + </span><span class="nl">"system"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"kafka"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"stream"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"topic-name"</span><span class="p">,</span><span class="w"> + </span><span class="nl">"partitionId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="w"> + </span><span class="p">}]</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span><span class="p">]</span></code></pre></figure> +<p>######Response codes</p> <table class="table table-condensed table-bordered table-striped"> <thead> <tr> @@ -696,38 +707,34 @@ messages of all of the tasks resource en </tr> <tr> <td>404 Not Found</td><td>Invalid job instance was provided as an argument. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Invalid arguments for getTasks. jobName: SamzaJobName jobId: SamzaJobId."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Invalid arguments for getTasks. jobName: SamzaJobName jobId: SamzaJobId."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> <tr> <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out. -<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span> - <span class="nt">"message"</span><span class="p">:</span> <span class="s2">"Timeout waiting for get all tasks."</span> -<span class="p">}</span></code></pre></figure> +<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> + </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Timeout waiting for get all tasks."</span><span class="w"> +</span><span class="p">}</span></code></pre></figure> </td> </tr> </tbody> </table> +<p><br /></p> -<p><br/></p> +<p><br /></p> -<p><br/></p> - -<h3 id="design">Design</h3> - -<h3 id="abstractions">Abstractions</h3> - -<p>There are two primary abstractions that are required by the TasksResource that users can implement to handle any details specific to their environment.</p> +<p>###Design +###Abstractions +There are two primary abstractions that are required by the TasksResource that users can implement to handle any details specific to their environment.</p> <ol> -<li> <strong>TaskProxy</strong>: This interface is the central point of interacting with Samza tasks. It exposes a method to get all the tasks of a Samza job.</li> -<li> <strong>InstallationFinder</strong>: The InstallationFinder provides a generic interface to discover all the installed jobs, hiding any customizations in the job package structure and its location. The InstallationFinder also resolves the job configuration, which is used to validate and identify the job.</li> + <li><strong>TaskProxy</strong>: This interface is the central point of interacting with Samza tasks. It exposes a method to get all the tasks of a Samza job.</li> + <li><strong>InstallationFinder</strong>: The InstallationFinder provides a generic interface to discover all the installed jobs, hiding any customizations in the job package structure and its location. The InstallationFinder also resolves the job configuration, which is used to validate and identify the job.</li> </ol> <h2 id="configuration">Configuration</h2> - <p>The TasksResource properties should be specified in the same file as the Samza REST configuration.</p> <table class="table table-condensed table-bordered table-striped"> @@ -745,7 +752,7 @@ messages of all of the tasks resource en <td>job.installations.path</td><td><b>Required:</b> The file system path which contains the Samza job installations. The path must be on the same host as the Samza REST Service. Each installation must be a directory with structure conforming to the expectations of the InstallationRecord implementation used by the JobProxy.</td> </tr> <tr> - <td>job.config.factory.class</td><td>The config factory to use for reading Samza job configs. This is used to fetch the job.name and job.id properties for each job instance in the InstallationRecord. It’s also used to validate that a particular directory within the installation path actually contains Samza jobs. If not specified <pre>org.apache.samza.config.factories.PropertiesConfigFactory</pre> will be used. </td> + <td>job.config.factory.class</td><td>The config factory to use for reading Samza job configs. This is used to fetch the job.name and job.id properties for each job instance in the InstallationRecord. It's also used to validate that a particular directory within the installation path actually contains Samza jobs. If not specified <pre>org.apache.samza.config.factories.PropertiesConfigFactory</pre> will be used. </td> </tr> </tbody> </table> Modified: samza/site/learn/documentation/latest/yarn/application-master.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/yarn/application-master.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/yarn/application-master.html (original) +++ samza/site/learn/documentation/latest/yarn/application-master.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/yarn/application-master">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/yarn/application-master">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/yarn/application-master">1.6.0</a></li> @@ -639,25 +653,25 @@ limitations under the License. --> -<p>YARN is Hadoop’s next-generation cluster manager. It allows developers to deploy and execute arbitrary commands on a grid. If you’re unfamiliar with YARN, or the concept of an ApplicationMaster (AM), please read Hadoop’s <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a> page.</p> +<p>YARN is Hadoopâs next-generation cluster manager. It allows developers to deploy and execute arbitrary commands on a grid. If youâre unfamiliar with YARN, or the concept of an ApplicationMaster (AM), please read Hadoopâs <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a> page.</p> <h3 id="integration">Integration</h3> -<p>Samza’s main integration with YARN comes in the form of a Samza ApplicationMaster. This is the chunk of code responsible for managing a Samza job in a YARN grid. It decides what to do when a stream processor fails, which machines a Samza job’s <a href="../container/samza-container.html">containers</a> should run on, and so on.</p> +<p>Samzaâs main integration with YARN comes in the form of a Samza ApplicationMaster. This is the chunk of code responsible for managing a Samza job in a YARN grid. It decides what to do when a stream processor fails, which machines a Samza jobâs <a href="../container/samza-container.html">containers</a> should run on, and so on.</p> <p>When the Samza ApplicationMaster starts up, it does the following:</p> <ol> -<li>Creates the <a href="../container/coordinator-stream.html#JobCoordinator">Job Coordinator</a> which bootstraps the Job Model and config from the <a href="../container/coordinator-stream.html">Coordinator Stream</a>.</li> -<li>Starts a JMX server on a random port.</li> -<li>Instantiates a metrics registry and reporters to keep track of relevant metrics.</li> -<li>Registers the AM with YARN’s RM.</li> -<li>Gets the total number of partitions for the Samza job using each input stream’s PartitionManager (see the <a href="../container/streams.html">Streams</a> page for details).</li> -<li>Read the total number of containers requested from the Samza job’s configuration.</li> -<li>Assign each partition to a container (called a Task Group in Samza’s AM dashboard).</li> -<li>Make a <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ResourceRequest.html";>ResourceRequest</a> to YARN for each container. If <a href="yarn-host-affinity.html">host-affinity</a> is enabled on the job, the AM uses the container locality information provided by the Job Coordinator and requests for the same host in the ResourceRequest.</li> -<li>Starts a ContainerAllocator thread that matches allocated containers and starts the container process.</li> -<li>Poll the YARN RM every second to check for allocated and released containers.</li> + <li>Creates the <a href="../container/coordinator-stream.html#JobCoordinator">Job Coordinator</a> which bootstraps the Job Model and config from the <a href="../container/coordinator-stream.html">Coordinator Stream</a>.</li> + <li>Starts a JMX server on a random port.</li> + <li>Instantiates a metrics registry and reporters to keep track of relevant metrics.</li> + <li>Registers the AM with YARNâs RM.</li> + <li>Gets the total number of partitions for the Samza job using each input streamâs PartitionManager (see the <a href="../container/streams.html">Streams</a> page for details).</li> + <li>Read the total number of containers requested from the Samza jobâs configuration.</li> + <li>Assign each partition to a container (called a Task Group in Samzaâs AM dashboard).</li> + <li>Make a <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ResourceRequest.html";>ResourceRequest</a> to YARN for each container. If <a href="yarn-host-affinity.html">host-affinity</a> is enabled on the job, the AM uses the container locality information provided by the Job Coordinator and requests for the same host in the ResourceRequest.</li> + <li>Starts a ContainerAllocator thread that matches allocated containers and starts the container process.</li> + <li>Poll the YARN RM every second to check for allocated and released containers.</li> </ol> <p>From this point on, the ApplicationMaster just reacts to events from the RM and delegates it to the ContainerAllocator thread.</p> @@ -666,17 +680,17 @@ <p>Whenever a container is allocated, the AM will work with the YARN NM to start a SamzaContainer (with appropriate partitions assigned to it) in the container. If a container fails with a non-zero return code, the AM will request a new container, and restart the SamzaContainer. If a SamzaContainer fails too many times, too quickly, the ApplicationMaster will fail the whole Samza job with a non-zero return code. See the yarn.container.retry.count and yarn.container.retry.window.ms <a href="../jobs/configuration.html">configuration</a> parameters for details.</p> -<p>When the AM receives a reboot signal from YARN, it will throw a SamzaException. This will trigger a clean and successful shutdown of the AM (YARN won’t think the AM failed).</p> +<p>When the AM receives a reboot signal from YARN, it will throw a SamzaException. This will trigger a clean and successful shutdown of the AM (YARN wonât think the AM failed).</p> -<p>If the AM, itself, fails, YARN will handle restarting the AM. When the AM is restarted, all containers that were running will be killed, and the AM will start from scratch. The same list of operations, shown above, will be executed. The AM will request new containers for its SamzaContainers, and proceed as though it has just started for the first time. YARN has a yarn.resourcemanager.am.max-retries configuration parameter that’s defined in <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml";>yarn-site.xml</a>. This configuration defaults to 1, which means that, by default, a single AM failure will cause your Samza job to stop running.</p> +<p>If the AM, itself, fails, YARN will handle restarting the AM. When the AM is restarted, all containers that were running will be killed, and the AM will start from scratch. The same list of operations, shown above, will be executed. The AM will request new containers for its SamzaContainers, and proceed as though it has just started for the first time. YARN has a yarn.resourcemanager.am.max-retries configuration parameter thatâs defined in <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml";>yarn-site.xml</a>. This configuration defaults to 1, which means that, by default, a single AM failure will cause your Samza job to stop running.</p> <h3 id="security">Security</h3> -<p>The Samza dashboard’s HTTP access is currently un-secured, even when using YARN in secure-mode. This means that users with access to a YARN grid could port-scan a Samza ApplicationMaster’s HTTP server, and open the dashboard in a browser to view its contents. Sensitive configuration can be viewed by anyone, in this way, and care should be taken. There are plans to secure Samza’s ApplicationMaster using <a href="http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.0/bk_installing_manually_book/content/rpm-chap14-2-3-1.html";>Hadoop’s security</a> features (<a href="http://en.wikipedia.org/wiki/SPNEGO";>SPENAGO</a>).</p> +<p>The Samza dashboardâs HTTP access is currently un-secured, even when using YARN in secure-mode. This means that users with access to a YARN grid could port-scan a Samza ApplicationMasterâs HTTP server, and open the dashboard in a browser to view its contents. Sensitive configuration can be viewed by anyone, in this way, and care should be taken. There are plans to secure Samzaâs ApplicationMaster using <a href="http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.0/bk_installing_manually_book/content/rpm-chap14-2-3-1.html";>Hadoopâs security</a> features (<a href="http://en.wikipedia.org/wiki/SPNEGO";>SPENAGO</a>).</p> -<p>See Samza’s <a href="../operations/security.html">security</a> page for more details.</p> +<p>See Samzaâs <a href="../operations/security.html">security</a> page for more details.</p> -<h2 id="isolation"><a href="isolation.html">Isolation »</a></h2> +<h2 id="isolation-"><a href="isolation.html">Isolation »</a></h2> </div> Modified: samza/site/learn/documentation/latest/yarn/isolation.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/yarn/isolation.html?rev=1906774&r1=1906773&r2=1906774&view=diff ============================================================================== --- samza/site/learn/documentation/latest/yarn/isolation.html (original) +++ samza/site/learn/documentation/latest/yarn/isolation.html Wed Jan 18 19:33:25 2023 @@ -227,6 +227,12 @@ + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> + + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> + + <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> @@ -538,6 +544,14 @@ + <li class="hide"><a href="/learn/documentation/1.8.0/yarn/isolation">1.8.0</a></li> + + + + <li class="hide"><a href="/learn/documentation/1.7.0/yarn/isolation">1.7.0</a></li> + + + <li class="hide"><a href="/learn/documentation/1.6.0/yarn/isolation">1.6.0</a></li> @@ -639,17 +653,17 @@ limitations under the License. --> -<p>When running Samza jobs in a shared, distributed environment, the stream processors can have an impact on one another’s performance. A stream processor that uses 100% of a machine’s CPU will slow down all other stream processors on the machine.</p> +<p>When running Samza jobs in a shared, distributed environment, the stream processors can have an impact on one anotherâs performance. A stream processor that uses 100% of a machineâs CPU will slow down all other stream processors on the machine.</p> -<p>One of <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a>‘s responsibilities is to manage resources so that this doesn’t happen. Each of YARN’s Node Managers (NM) has a chunk of “resources” dedicated to it. The YARN Resource Manager (RM) will only allow a container to be allocated on a NM if it has enough resources to satisfy the container’s needs.</p> +<p>One of <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a>âs responsibilities is to manage resources so that this doesnât happen. Each of YARNâs Node Managers (NM) has a chunk of âresourcesâ dedicated to it. The YARN Resource Manager (RM) will only allow a container to be allocated on a NM if it has enough resources to satisfy the containerâs needs.</p> <p>YARN currently supports resource management for memory and CPU.</p> <h3 id="memory">Memory</h3> -<p>YARN will automatically enforce memory limits for all containers that it executes. All containers must have a max-memory size defined when they’re created. If the sum of all memory usage for processes associated with a single YARN container exceeds this maximum, YARN will kill the container.</p> +<p>YARN will automatically enforce memory limits for all containers that it executes. All containers must have a max-memory size defined when theyâre created. If the sum of all memory usage for processes associated with a single YARN container exceeds this maximum, YARN will kill the container.</p> -<p>Samza supports memory limits using the yarn.container.memory.mb and yarn.am.container.memory.mb configuration parameters. Keep in mind that this is simply the amount of memory YARN will allow a <a href="../container/samza-container.html">SamzaContainer</a> or <a href="application-master.html">ApplicationMaster</a> to have. You’ll still need to configure your heap settings appropriately using task.opts, when using Java (the default is -Xmx160M). See the <a href="../jobs/configuration.html">Configuration</a> and <a href="../jobs/packaging.html">Packaging</a> pages for details.</p> +<p>Samza supports memory limits using the yarn.container.memory.mb and yarn.am.container.memory.mb configuration parameters. Keep in mind that this is simply the amount of memory YARN will allow a <a href="../container/samza-container.html">SamzaContainer</a> or <a href="application-master.html">ApplicationMaster</a> to have. Youâll still need to configure your heap settings appropriately using task.opts, when using Java (the default is -Xmx160M). See the <a href="../jobs/configuration.html">Configuration</a> and <a href="../jobs/packaging.html">Packaging</a> pages for details.</p> <h3 id="cpu">CPU</h3> @@ -657,13 +671,13 @@ <h4 id="cgroups">CGroups</h4> -<p>Unlike memory, which YARN can enforce itself (by looking at the /proc folder), YARN can’t enforce CPU isolation, since this must be done at the Linux kernel level. One of YARN’s interesting new features is its support for Linux <a href="https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt";>CGroups</a>. CGroups are a way to control process utilization at the kernel level in Linux.</p> +<p>Unlike memory, which YARN can enforce itself (by looking at the /proc folder), YARN canât enforce CPU isolation, since this must be done at the Linux kernel level. One of YARNâs interesting new features is its support for Linux <a href="https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt";>CGroups</a>. CGroups are a way to control process utilization at the kernel level in Linux.</p> -<p>If YARN is setup to use CGroups, then YARN will guarantee that a container will get at least the amount of CPU that it requires. Currently, YARN will give you more CPU, if it’s available. For details on enforcing “at most” CPU usage, see <a href="https://issues.apache.org/jira/browse/YARN-810";>YARN-810</a>.</p> +<p>If YARN is setup to use CGroups, then YARN will guarantee that a container will get at least the amount of CPU that it requires. Currently, YARN will give you more CPU, if itâs available. For details on enforcing âat mostâ CPU usage, see <a href="https://issues.apache.org/jira/browse/YARN-810";>YARN-810</a>.</p> <p>See <a href="http://riccomini.name/posts/hadoop/2013-06-14-yarn-with-cgroups/";>this blog post</a> for details on setting up YARN with CGroups.</p> -<h2 id="host-affinity-yarn"><a href="../yarn/yarn-host-affinity.html">Host Affinity & YARN »</a></h2> +<h2 id="host-affinity--yarn-"><a href="../yarn/yarn-host-affinity.html">Host Affinity & YARN »</a></h2> </div>
