[ https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=382547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382547 ]
ASF GitHub Bot logged work on BEAM-9219: ---------------------------------------- Author: ASF GitHub Bot Created on: 05/Feb/20 21:47 Start Date: 05/Feb/20 21:47 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10745: [BEAM-9219] Streamline creation of Python and Java dependencies pages URL: https://github.com/apache/beam/pull/10745#discussion_r375526685 ########## File path: website/src/documentation/sdks/python-dependencies.md ########## @@ -26,460 +26,41 @@ behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution. -<p>To see the compile and runtime dependencies for your Beam SDK version, expand -the relevant section below.</p> +Dependencies for your Beam SDK version are listed in `setup.py` in the Beam repository. To view them, perform the following steps: -<details><summary markdown="span"><b>2.17.0</b></summary> +1. Open `setup.py`. -<p>Beam SDK for Python 2.17.0 has the following compile and runtime dependencies.</p> + ``` + https://raw.githubusercontent.com/apache/beam/v<version_number>/sdks/python/setup.py + ``` + + Replace `<version_number>` with the major.minor.patch version of the SDK. For example, <https://raw.githubusercontent.com/apache/beam/v2.18.0/sdks/python/setup.py>{:target="_blank"} will provide the dependencies for the 2.18.0 release. + + +2. Review the core dependency list under `REQUIRED_PACKAGES`. -<table class="table-bordered table-striped"> -<tr><th>Package</th><th>Version</th></tr> - <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr> - <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr> - <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr> - <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr> - <tr><td>dill</td><td>>=0.3.0,<0.3.1</td></tr> - <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr> - <tr><td>funcsigs</td><td>>=1.0.2,<2; python_version < "3.0"</td></tr> - <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr> - <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr> - <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr> - <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr> - <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<1.1.0</td></tr> - <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr> - <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr> - <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<1.1.0</td></tr> - <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr> - <tr><td>grpcio</td><td>>=1.12.1,<2</td></tr> - <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr> - <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr> - <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr> - <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr> - <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr> - <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr> - <tr><td>pyarrow</td><td>>=0.15.1,<0.16.0; python_version >= "3.0" or platform_system != "Windows"</td></tr> - <tr><td>pydot</td><td>>=1.2.0,<2</td></tr> - <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr> - <tr><td>python-dateutil</td><td>>=2.8.0,<3</td></tr> - <tr><td>pytz</td><td>>=2018.3</td></tr> - <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr> - <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr> -</table> + **Note:** If you require [extra features]({{ site.baseurl }}/get-started/quickstart-py#extra-requirements) such as `gcp` or `test`, you should review the lists under `REQUIRED_TEST_PACKAGES`, `GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies. -</details> +You can also retrieve the dependency list from the command line using the following process: -<details><summary markdown="span"><b>2.16.0</b></summary> +1. Create a clean virtual environment on your local machine. -<p>Beam SDK for Python 2.16.0 has the following compile and - runtime dependencies.</p> + Python 3: -<table class="table-bordered table-striped"> - <tr><th>Package</th><th>Version</th></tr> - <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr> - <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr> - <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr> - <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr> - <tr><td>dill</td><td>>=0.3.0,<0.3.1</td></tr> - <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr> - <tr><td>funcsigs</td><td>>=1.0.2,<2; python_version < "3.0"</td></tr> - <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr> - <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr> - <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr> - <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr> - <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<1.1.0</td></tr> - <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr> - <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr> - <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<1.1.0</td></tr> - <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr> - <tr><td>grpcio</td><td>>=1.12.1,<2</td></tr> - <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr> - <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr> - <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr> - <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr> - <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr> - <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr> - <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr> - <tr><td>pydot</td><td>>=1.2.0,<2</td></tr> - <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr> - <tr><td>python-dateutil</td><td>>=2.8.0,<3</td></tr> - <tr><td>pytz</td><td>>=2018.3</td></tr> - <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr> - <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr> - <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr> -</table> + ``` + $ python3 -m venv env && source env/bin/activate + ``` + + Python 2: -</details> + ``` + $ pip install virtualenv && virtualenv env && source env/bin/activate + ``` -<details><summary markdown="span"><b>2.15.0</b></summary> +2. [Install the Beam Python SDK]({{ site.baseurl }}/get-started/quickstart-py/#download-and-install). Review comment: I actually asked in an earlier comment to link existing installation instruction to keep all things related to installation in one place, and not duplicate these instructions on multiple pages. For example https://beam.apache.org/get-started/quickstart-py/#download-and-install mentions how to install beam with [gcp] deps, etc. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 382547) Time Spent: 4h 40m (was: 4.5h) > Streamline creation of Python and Java dependencies pages > --------------------------------------------------------- > > Key: BEAM-9219 > URL: https://issues.apache.org/jira/browse/BEAM-9219 > Project: Beam > Issue Type: Improvement > Components: website > Reporter: David Wrede > Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > This issue is about the need to address keeping both Python and Java SDK > dependency pages more relevant and up-to-date while reducing the amount of > time it takes to provide that information. The current method of scraping and > copying dependencies into a table for every release is a non-trivial task > because of the semi-automated workflows done by the tech writers on the > website. > In an effort to provide accurate dependency listings that are always in sync > with SDK releases, referring people to the appropriate places in the source > code (or through CLI commands) should provide people the information they are > looking for and not require the creation and maintenance of an automated > tooling solution to generate the dependency tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)