This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 3612d22b164 Publishing website 2023/04/12 22:15:39 at commit 942bf87 3612d22b164 is described below commit 3612d22b164f04ad9de143e1ab81a20439b84120 Author: jenkins <bui...@apache.org> AuthorDate: Wed Apr 12 22:15:40 2023 +0000 Publishing website 2023/04/12 22:15:39 at commit 942bf87 --- .../documentation/sdks/python-pipeline-dependencies/index.html | 8 ++++++-- website/generated-content/sitemap.xml | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html index da2d998111c..d771e9884af 100644 --- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html +++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html @@ -20,7 +20,7 @@ function addPlaceholder(){$('input:text').attr('placeholder',"What are you looking for?");} function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");} function blockScroll(){$("body").toggleClass("fixedPosition");} -function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Languages</span></li><li><span class=section-nav-list-title>Java</span><ul class=section-nav-list><li><a href=/documentation/sdks/java/>Java SDK overvi [...] +function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Languages</span></li><li><span class=section-nav-list-title>Java</span><ul class=section-nav-list><li><a href=/documentation/sdks/java/>Java SDK overvi [...] </code></pre><p>This command creates a <code>requirements.txt</code> file that lists all packages that are installed on your machine, regardless of where they were installed from.</p></li><li><p>Edit the <code>requirements.txt</code> file and delete all packages that are not relevant to your code.</p></li><li><p>Run your pipeline with the following command-line option:</p><pre><code> --requirements_file requirements.txt </code></pre><p>The runner will use the <code>requirements.txt</code> file to install your additional dependencies onto the remote workers.</p></li></ol><blockquote><p><strong>NOTE</strong>: An alternative to <code>pip freeze</code> is to use a library like <a href=https://github.com/jazzband/pip-tools>pip-tools</a> to compile all the dependencies required for the pipeline from a <code>--requirements_file</code>, where only top-level dependencies are mentioned.</p></blockquote><h2 id=cus [...] COPY <path to requirements.txt> /tmp/requirements.txt @@ -48,7 +48,11 @@ package then you can build the tarball with the following command:</p><pre><code </code></pre><p>See the <a href=https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset>Juliaset</a> project for an example that follows this required project structure.</p></li><li><p>Run your pipeline with the following command-line option:</p><pre><code> --setup_file /path/to/setup.py </code></pre></li></ol><p><strong>Note:</strong> Because custom commands execute after the dependencies for your workflow are installed (by <code>pip</code>), you should omit the PyPI package dependency from the pipeline’s <code>requirements.txt</code> file and from the <code>install_requires</code> parameter in the <code>setuptools.setup()</code> call of your <code>setup.py</code> file.</p><h2 id=pre-building-sdk-container-image>Pre-building SDK container image</h2><p>In pipeline [...] However, it may be possible to pre-build the SDK containers and perform the dependency installation once before the workers start with <code>--prebuild_sdk_container_engine</code>. For instructions of how to use pre-building with Google Cloud -Dataflow, see <a href=https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild>Pre-building the python SDK custom container image with extra dependencies</a>.</p><p><strong>NOTE</strong>: This feature is available only for the <code>Dataflow Runner v2</code>.</p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col__logo><img src=/images/beam_logo_cir [...] +Dataflow, see <a href=https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild>Pre-building the python SDK custom container image with extra dependencies</a>.</p><p><strong>NOTE</strong>: This feature is available only for the <code>Dataflow Runner v2</code>.</p><h2 id=pickling-and-managing-main-session>Pickling and Managing Main Session</h2><p>Pickling in the Python SDK is set up to pickle the state of the global namespace. By default, global imports, functions, an [...] +Thus, one might encounter an unexpected <code>NameError</code> when running a <code>DoFn</code> on any remote runner using portability. To resolve this, manage the main session by +simply setting the main session. This will load the pickled state of the global namespace onto the Dataflow workers. +For example, see <a href=https://cloud.google.com/dataflow/docs/guides/common-errors#how-do-i-handle-nameerrors>Handling NameErrors</a> to set the main session on the <code>DataflowRunner</code>.</p><p><strong>NOTE</strong>: This applies to the Python SDK executing with the dill pickler on any remote runner using portability. Therefore, this issue will +not occur in <code>DirectRunner</code>. Additionally, note that the dill pickler is the default pickler in the Python SDK.</p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg class=footer__logo alt="Apache logo [...] <a href=https://www.apache.org>The Apache Software Foundation</a> | <a href=/privacy_policy>Privacy Policy</a> | <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col--group><div class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...] \ No newline at end of file diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 4ef93246939..fb274f8e4e8 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.46.0/</loc><lastmod>2023-04-12T18:12:36+03:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-04-12T18:12:36+03:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-04-12T18:12:36+03:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-04-12T18:12:36+03:00</lastmod></url><url><loc>/catego [...] \ No newline at end of file +<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.46.0/</loc><lastmod>2023-04-12T16:30:14-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-04-12T16:30:14-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-04-12T16:30:14-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-04-12T16:30:14-04:00</lastmod></url><url><loc>/catego [...] \ No newline at end of file