[GitHub] [airflow] mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment
mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment URL: https://github.com/apache/airflow/pull/6590#discussion_r347112557 ## File path: airflow/gcp/hooks/dataflow.py ## @@ -515,8 +530,20 @@ def label_formatter(labels_dict): return ['--labels={}={}'.format(key, value) for key, value in labels_dict.items()] -self._start_dataflow(variables, name, [py_interpreter] + py_options + [dataflow], - label_formatter, project_id) +if py_requirements is not None: +with TemporaryDirectory(prefix='dataflow-venv') as tmp_dir: +py_interpreter = prepare_virtualenv( Review comment: 1. If you want to set up a private pip repository for Cloud Composer, then you should follow these instructions: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-private You will also need to set the ``PIP_CONFIG_FILE`` environment variable when you use the virtual environment 2. Unfortunately, in distributed systems we should not save files on the current environment and rely on the fact that they will be available on restart. This can cause many problems, e.g. taking up all the disk space. Instead, I recommend delivering packages from a local folder that you have previously set up. I have prepared a script that shows how to download all packages to the directory and then install them in a virtual environment ```bash #!/bin/bash -x TMP_PYPI="$(mktemp -d)" TMP_VENV="$(mktemp -d)" virtualenv ${TMP_PYPI}/venv virtualenv ${TMP_VENV} START_FETCH=$(date +%s) source ${TMP_VENV}/bin/activate mkdir -p "${TMP_PYPI}/packages" pip download apache-beam[gcp] -d "${TMP_PYPI}/packages" END_FETCH=$(date +%s) START_INSTALL=$(date +%s) source ${TMP_VENV}/bin/activate pip install $(find "${TMP_PYPI}/packages" -type f) END_INSTALL=$(date +%s) echo "Fetch time: $(( END_FETCH - START_FETCH )) seconds" echo "Install time: $(( END_INSTALL - START_INSTALL )) seconds" ``` For me, the download process took a long time, but the local installation was significantly faster. ``` Fetch time: 41 seconds Install time: 20 seconds ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment
mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment URL: https://github.com/apache/airflow/pull/6590#discussion_r347112557 ## File path: airflow/gcp/hooks/dataflow.py ## @@ -515,8 +530,20 @@ def label_formatter(labels_dict): return ['--labels={}={}'.format(key, value) for key, value in labels_dict.items()] -self._start_dataflow(variables, name, [py_interpreter] + py_options + [dataflow], - label_formatter, project_id) +if py_requirements is not None: +with TemporaryDirectory(prefix='dataflow-venv') as tmp_dir: +py_interpreter = prepare_virtualenv( Review comment: 1. If you want to set up a private pip repository for Cloud Composer, then you should follow these instructions: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-private You will also need to set the ``PIP_CONFIG_FILE`` environment variable when you use the virtual environment 2. Unfortunately, in distributed systems we should not save files on the current environment and rely on the fact that they will be available on restart. This can cause many problems, e.g. taking up all the disk space. Instead, I recommend delivering packages from a local folder that you have previously set up. I have prepared a script that shows how to download all packages to the directory and then install them in a virtual environment ```bash #!/bin/bash -x TMP_PYPI="$(mktemp -d)" TMP_VENV="$(mktemp -d)" virtualenv ${TMP_PYPI}/venv virtualenv ${TMP_VENV} START_FETCH=$(date +%s) source ${TMP_VENV}/bin/activate mkdir -p "${TMP_PYPI}/packages" pip download apache-beam[gcp] -d "${TMP_PYPI}/packages" END_FETCH=$(date +%s) START_INSTALL=$(date +%s) source ${TMP_VENV}/bin/activate pip install $(find "${TMP_PYPI}/packages" -type f) END_INSTALL=$(date +%s) echo "Fetch time: $(( END_FETCH - START_FETCH )) seconds" echo "Install time: $(( END_INSTALL - START_INSTALL )) seconds" ``` For me, the download process took a long time, but the local installation was significantly faster. ``` Fetch time: 41 seconds Install time: 20 seconds ``` It is worth remembering that just starting the workers takes a few minutes, so in this case the differences of a few seconds will not matter much. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment
mik-laj commented on a change in pull request #6590: [AIRFLOW-5520] Add options to run Dataflow in a virtual environment URL: https://github.com/apache/airflow/pull/6590#discussion_r347680244 ## File path: airflow/gcp/hooks/dataflow.py ## @@ -481,30 +483,43 @@ def start_python_dataflow( variables: Dict, dataflow: str, py_options: List[str], +py_interpreter: str = "python2", +py_requirements: Optional[List[str]] = None, +py_system_site_packages: bool = False, project_id: Optional[str] = None, append_job_name: bool = True, -py_interpreter: str = "python2" ): """ Starts Dataflow job. :param job_name: The name of the job. :type job_name: str :param variables: Variables passed to the job. -:type variables: dict +:type variables: Dict :param dataflow: Name of the Dataflow process. :type dataflow: str :param py_options: Additional options. -:type py_options: list -:param append_job_name: True if unique suffix has to be appended to job name. -:type append_job_name: bool -:param project_id: Optional, the GCP project ID in which to start a job. -If set to None or missing, the default project_id from the GCP connection is used. +:type py_options: List[str] :param py_interpreter: Python version of the beam pipeline. If None, this defaults to the python2. To track python versions supported by beam and related issues check: https://issues.apache.org/jira/browse/BEAM-1251 +:param py_requirements: Additional python package to install. +If a value is passed to this parameter, a new virtual environment has been created with +additional packages installed. + +You could also install the apache-beam package if it is not installed on your system or you want +to use a different version. +:type py_requirements: List[str] +:param py_system_site_packages: Whether to include system_site_packages in your virtualenv. +See virtualenv documentation for more information. + +This option is only relevant if the AAA parameter is passed. Review comment: Fixed. Thanks AAA=py_requirements. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services