[GitHub] XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc
XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#issuecomment-424966969 Hi bad @ashb who didn't write test, may you have another look? ;-) The a few points you highlighted are covered: - Patterns will not only be checked for file name itself. The path will also be one part of the pattern matching (this is also reflected in my example) - Each subfolder can have its own `.airflowignore` file. This is described in the last paragraph when I introduce the scope of `.airflowignore` file. I also tried to run some local tests and confirmed it's working as expected. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JacobHayes commented on a change in pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code
JacobHayes commented on a change in pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code URL: https://github.com/apache/incubator-airflow/pull/3952#discussion_r220782677 ## File path: docs/howto/write-logs.rst ## @@ -89,54 +89,21 @@ Writing Logs to Google Cloud Storage Follow the steps below to enable Google Cloud Storage logging. -#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended. -#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``. -#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above. -#. Customize the following portions of the template: - -.. code-block:: bash - -# Add this variable to the top of the file. Note the trailing slash. -GCS_LOG_FOLDER = 'gs:///' - -# Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG -LOGGING_CONFIG = ... - -# Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable -'gcs.task': { -'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler', -'formatter': 'airflow.task', -'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER), -'gcs_log_folder': GCS_LOG_FOLDER, -'filename_template': FILENAME_TEMPLATE, -}, - -# Update the airflow.task and airflow.task_runner blocks to be 'gcs.task' instead of 'file.task'. -'loggers': { -'airflow.task': { -'handlers': ['gcs.task'], -... -}, -'airflow.task_runner': { -'handlers': ['gcs.task'], -... -}, -'airflow': { -'handlers': ['console'], -... -}, -} - -#. Make sure a Google Cloud Platform connection hook has been defined in Airflow. The hook should have read and write access to the Google Cloud Storage bucket defined above in ``GCS_LOG_FOLDER``. - -#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain: +To enable this feature, ``airflow.cfg`` must be configured as in this +example: -.. code-block:: bash +.. code-block:: bash -task_log_reader = gcs.task -logging_config_class = log_config.LOGGING_CONFIG -remote_log_conn_id = +[core] +# Airflow can store logs remotely in AWS S3. Users must supply a remote +# location URL (starting with either 's3://...') and an Airflow connection +# id that provides access to the storage location. +remote_logging_enabled = True Review comment: This variable should be `remote_logging` https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/airflow/config_templates/airflow_local_settings.py#L173 Also, the comment above this line should probably be changed to reference GCS (and `gs://`) instead of S3. Other than that, this seems to match my working setup with airflow GCS logging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp closed pull request #3959: [WIP] Backfill unit test for postgres operator.
jmcarp closed pull request #3959: [WIP] Backfill unit test for postgres operator. URL: https://github.com/apache/incubator-airflow/pull/3959 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/tests/operators/test_postgres_operator.py b/tests/operators/test_postgres_operator.py new file mode 100644 index 00..9a1dc5db7e --- /dev/null +++ b/tests/operators/test_postgres_operator.py @@ -0,0 +1,51 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import unittest + +from airflow.operators.postgres_operator import PostgresOperator + +try: +from unittest import mock +except ImportError: +try: +import mock +except ImportError: +mock = None + + +class TestPostgresOperator(unittest.TestCase): + +@mock.patch('airflow.operators.postgres_operator.PostgresHook') +def test_execute(self, mock_hook): +operator = PostgresOperator( +task_id='postgres', +sql='select 1', +) + +operator.execute(None) +mock_hook.assert_called_once_with( +postgres_conn_id='postgres_default', +schema=None, +) +mock_hook.return_value.run.assert_called_once_with( +'select 1', +False, +parameters=None, +) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629663#comment-16629663 ] ASF GitHub Bot commented on AIRFLOW-668: stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3 URL: https://github.com/apache/incubator-airflow/pull/1918 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Configuration parsing doesn't work properly with python 3 > - > > Key: AIRFLOW-668 > URL: https://issues.apache.org/jira/browse/AIRFLOW-668 > Project: Apache Airflow > Issue Type: Bug > Environment: Airflow version: v1.7.1.3 > - Airflow components: webserver and scheduler with a postgres database and > CeleryExecutor > - Python Version: 3.4.5 >Reporter: Rafael Gomes Fernandes >Assignee: Rafael Gomes Fernandes >Priority: Major > > The problem is: if you use python3 and the '_cmd' on the config file airflow > will not start due the error: > {noformat} > File > "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", > line 447, in _validate > "sqlite" in self.get('core', 'sql_alchemy_conn')): > TypeError: 'str' does not support the buffer interface > {noformat} > To reproduce the problem change the following line on airflow.cfg: > {code:title=airflow.cfg|borderStyle=solid} > sql_alchemy_conn_cmd = echo sqlite:~/airflow/airflow.db > {code} > The solution is change the following run_command method's line on > airflow/configuration.py: > {code:title=airflow/configuration.py|borderStyle=solid} > command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, > universal_newlines=True) > {code} > By setting the universal_newlines to true the file objects stdout and stderr > are opened as text files and treated as string in python 2 and python 3 > avoiding the error. > run_command with universal_newlines=True: > When using python 3 output type: and no error. > When using python 2 output type: and no error. > run_command as it is : > When using python 3 output type: and TypeError. > When using python 2 output type: and no error. > I have tested the change with Travis CI and it passed, but when I tried to > run tox or unit test I found this problems: > {code:title=run_unit_tests.sh|borderStyle=solid} > ./run_unit_tests.sh -q -c airflow.cfg > Initializing the DB > Starting the unit tests with the following nose arguments: -q -c airflow.cfg > == > ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: > task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = > ?'] [parameters: ('unit_tests',)]) > -- > Traceback (most recent call last): > File > "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py", > line 39, in runTest > raise self.exc_val.with_traceback(self.tb) > File > "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py", > line 418, in loadTestsFromName > addr.filename, addr.module) > File > "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", > line 47, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", > line 94, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in > load_module > return load_package(name, filename) > File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in > load_package > return methods.load() > File "", line 1220, in load > File "", line 1200, in _load_unlocked > File "", line 1129, in _exec > File "", line 1471, in exec_module > File "", line 321, in _call_with_frames_removed > File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in > > from .contrib import * > File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line > 16, in > from .operators import * > File > "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py", > line 17, in > from .ssh_execute_operator import * > File > "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", > line 37, in > reset() > File > "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", > line 33,
[jira] [Commented] (AIRFLOW-419) Setting task instance status through UI causes exception
[ https://issues.apache.org/jira/browse/AIRFLOW-419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629664#comment-16629664 ] ASF GitHub Bot commented on AIRFLOW-419: stale[bot] closed pull request #2537: [AIRFLOW-419] Fix microsecond-precision datetime decode issue URL: https://github.com/apache/incubator-airflow/pull/2537 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Setting task instance status through UI causes exception > > > Key: AIRFLOW-419 > URL: https://issues.apache.org/jira/browse/AIRFLOW-419 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.7.1 > Environment: Using the Celery executor and PostgreSQL as the backing > DB. >Reporter: Mjumbe Poe >Priority: Major > > When using postgres, datetimes are returned down to the microsecond, and > breaks the datetime parsing in > {{TaskInstanceModelView.set_task_instance_state}} at > https://github.com/mjumbewu/incubator-airflow/blob/55985ef/airflow/www/views.py#L2252. > This blocks users from updating the task instance status through the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-428) Scheduler restart (via SIGKILL) leaves orphaned celery and airflow run processes
[ https://issues.apache.org/jira/browse/AIRFLOW-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629665#comment-16629665 ] ASF GitHub Bot commented on AIRFLOW-428: stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM URL: https://github.com/apache/incubator-airflow/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler restart (via SIGKILL) leaves orphaned celery and airflow run > processes > > > Key: AIRFLOW-428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-428 > Project: Apache Airflow > Issue Type: Bug >Reporter: Li Xuanji >Assignee: Li Xuanji >Priority: Major > > Airflow worker currently becomes a celery process. Sending a SIGTERM to it > performs a warm shutdown which waits for tasks to exit. We use runit which > will send a SIGKILL after some time passes; the SIGKILL causes the airflow > run commands to be orphaned (parented onto PID 1 init, celery does not know > about them). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] stale[bot] closed pull request #2490: User should be able to decide TemporaryDirectory root in a BashOperator
stale[bot] closed pull request #2490: User should be able to decide TemporaryDirectory root in a BashOperator URL: https://github.com/apache/incubator-airflow/pull/2490 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3
stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3 URL: https://github.com/apache/incubator-airflow/pull/1918 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #2666: Updating column header in table to match column.
stale[bot] closed pull request #2666: Updating column header in table to match column. URL: https://github.com/apache/incubator-airflow/pull/2666 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM
stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM URL: https://github.com/apache/incubator-airflow/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp opened a new pull request #3959: [WIP] Backfill unit test for postgres operator.
jmcarp opened a new pull request #3959: [WIP] Backfill unit test for postgres operator. URL: https://github.com/apache/incubator-airflow/pull/3959 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode
akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode URL: https://github.com/apache/incubator-airflow/pull/3958 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-3124) issues and references them in the PR title. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: The command `airflow webserver -d` crashes when `settings.RBAC == True`. This is because `create_app` currently returns `app, appbuilder`, which causes the app to crash with an `AttributeError`. Reading through the code, it doesn't look like we rely on `appbuilder` being returned by `create_app` anywhere, so it should be safe to remove this. ### Tests - [x] My PR adds the following unit tests I set up the CLI to pass in the `unit_test_mode` setting to the flask app so we can disable the werkzeug reloader when running tests. If the reloader runs, there is an extra monitoring process left running after killing the main webserver process. This causes conflicts if multiple tests rely on being able to start a webserver (as the process will not be able to bind to the default socket). ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3124) Broken webserver debug mode (RBAC)
[ https://issues.apache.org/jira/browse/AIRFLOW-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629528#comment-16629528 ] ASF GitHub Bot commented on AIRFLOW-3124: - akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode URL: https://github.com/apache/incubator-airflow/pull/3958 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-3124) issues and references them in the PR title. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: The command `airflow webserver -d` crashes when `settings.RBAC == True`. This is because `create_app` currently returns `app, appbuilder`, which causes the app to crash with an `AttributeError`. Reading through the code, it doesn't look like we rely on `appbuilder` being returned by `create_app` anywhere, so it should be safe to remove this. ### Tests - [x] My PR adds the following unit tests I set up the CLI to pass in the `unit_test_mode` setting to the flask app so we can disable the werkzeug reloader when running tests. If the reloader runs, there is an extra monitoring process left running after killing the main webserver process. This causes conflicts if multiple tests rely on being able to start a webserver (as the process will not be able to bind to the default socket). ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Broken webserver debug mode (RBAC) > -- > > Key: AIRFLOW-3124 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3124 > Project: Apache Airflow > Issue Type: Bug > Components: webapp, webserver >Reporter: Aaron Kosel >Assignee: Aaron Kosel >Priority: Minor > > {code:java} > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 7, in > exec(compile(f.read(), __file__, 'exec')) > File "/airflow/airflow/bin/airflow", line 32, in > args.func(args) > File "/airflow/airflow/utils/cli.py", line 74, in wrapper > return f(*args, **kwargs) > File "/airflow/airflow/bin/cli.py", line 875, in webserver > app.run(debug=True, port=args.port, host=args.hostname, > AttributeError: 'tuple' object has no attribute 'run' > {code} > Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204, > but only affecting RBAC debug mode. The problem is that `create_app` returns > a tuple, but the `cli` script expects to just receive the flask app back > without the appbuilder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3124) Broken webserver debug mode (RBAC)
Aaron Kosel created AIRFLOW-3124: Summary: Broken webserver debug mode (RBAC) Key: AIRFLOW-3124 URL: https://issues.apache.org/jira/browse/AIRFLOW-3124 Project: Apache Airflow Issue Type: Bug Components: webapp, webserver Reporter: Aaron Kosel Assignee: Aaron Kosel {code:java} Traceback (most recent call last): File "/usr/local/bin/airflow", line 7, in exec(compile(f.read(), __file__, 'exec')) File "/airflow/airflow/bin/airflow", line 32, in args.func(args) File "/airflow/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File "/airflow/airflow/bin/cli.py", line 875, in webserver app.run(debug=True, port=args.port, host=args.hostname, AttributeError: 'tuple' object has no attribute 'run' {code} Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204, but only affecting RBAC debug mode. The problem is that `create_app` returns a tuple, but the `cli` script expects to just receive the flask app back without the appbuilder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] newtonle commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#discussion_r220743204 ## File path: airflow/models.py ## @@ -3389,7 +3389,7 @@ def __init__( self.on_success_callback = on_success_callback self.on_failure_callback = on_failure_callback -self._context_manager_set = False +self._old_context_manager_dags = [] Review comment: Actually this doesn't contain the current one. It just contains what was in `_CONTEXT_MANAGER_DAG` before it is set to the current DAG. The `old` naming convention was existing, and the only difference here is making into a list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aoen commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
aoen commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#discussion_r220735705 ## File path: airflow/models.py ## @@ -3389,7 +3389,7 @@ def __init__( self.on_success_callback = on_success_callback self.on_failure_callback = on_failure_callback -self._context_manager_set = False +self._old_context_manager_dags = [] Review comment: Nit: How about just context_manager_dags since it includes the current one? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dimberman commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
dimberman commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-424881819 +1 thank you @kaxil! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2952) Dockerized CI pipeline has silently broken integration testing for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629441#comment-16629441 ] ASF GitHub Bot commented on AIRFLOW-2952: - kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. - https://issues.apache.org/jira/browse/AIRFLOW-2952 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: - Update outdated cli command to create user - Remove `airflow/example_dags_kubernetes` as the dag already exists in `contrib/example_dags/` - Update the path to copy K8s dags ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Dockerized CI pipeline has silently broken integration testing for > KubernetesExecutor > - > > Key: AIRFLOW-2952 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2952 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 2.0.0 > > > [~gcuriel] [~bolke] [~Fokko] > Looking at all recent builds the new CI pipeline is silently reverting the > kubernetes tests to the normal airflow tests. > Before > https://travis-ci.org/apache/incubator-airflow/jobs/418914949#L1007 > After: > [https://travis-ci.org/apache/incubator-airflow/jobs/419062412#L4970] > This means that kubernetes builds will pass without actually testing on a > kubernetes cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI
kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. - https://issues.apache.org/jira/browse/AIRFLOW-2952 ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: - Update outdated cli command to create user - Remove `airflow/example_dags_kubernetes` as the dag already exists in `contrib/example_dags/` - Update the path to copy K8s dags ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] newtonle commented on issue #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle commented on issue #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#issuecomment-424878741 @aoen This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3123) DAG context manager fails in nested cases
[ https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629440#comment-16629440 ] ASF GitHub Bot commented on AIRFLOW-3123: - newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3123 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Implemented a stack to manage the DAG context. Entering a DAG context pushes the past context on the stack. Exiting pops the stack to reset the context. This allows arbitrary nesting of contexts, particularly if the same DAG object is used in the nesting. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Modified `test_dag_as_context_manager` in `models.py` to capture this condition. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DAG context manager fails in nested cases > - > > Key: AIRFLOW-3123 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3123 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Reporter: Newton Le >Assignee: Newton Le >Priority: Major > > DAG context manager fails under some cases with nested contexts: > {code:python} > with DAG( ... ) as dag: > op1 = Operator() > with dag: > op2 = Operator() > op3 = Operator > {code} > op3 will not continue to be assigned the original DAG after exiting the > nested context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3123 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: Implemented a stack to manage the DAG context. Entering a DAG context pushes the past context on the stack. Exiting pops the stack to reset the context. This allows arbitrary nesting of contexts, particularly if the same DAG object is used in the nesting. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Modified `test_dag_as_context_manager` in `models.py` to capture this condition. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3123) DAG context manager fails in nested cases
Newton Le created AIRFLOW-3123: -- Summary: DAG context manager fails in nested cases Key: AIRFLOW-3123 URL: https://issues.apache.org/jira/browse/AIRFLOW-3123 Project: Apache Airflow Issue Type: Bug Components: DAG Reporter: Newton Le Assignee: Newton Le DAG context manager fails under some cases with nested contexts: {code:python} with DAG( ... ) as dag: op1 = Operator() with dag: op2 = Operator() op3 = Operator {code} op3 will not continue to be assigned the original DAG after exiting the nested context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629431#comment-16629431 ] Kaxil Naik commented on AIRFLOW-3118: - I know the reason probably. Can you try and upload any DAG file in ~/airflow/dags folder and restart both webserver and scheduler? > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: Screenshot_20180926_161837.png, > image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[ https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Smith Mathieu reassigned AIRFLOW-3036: -- Assignee: (was: Iuliia Volkova) > Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL > > > Key: AIRFLOW-3036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3036 > Project: Apache Airflow > Issue Type: Bug > Components: core, db >Affects Versions: 1.10.0 > Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow > 1.10 on Debian Stretch, Google Cloud SQL MySQL >Reporter: Smith Mathieu >Priority: Blocker > Labels: 1.10, google, google-cloud-sql > Fix For: 2.0.0 > > > The upgrade path to airflow 1.10 seems impossible for users of MySQL in > Google's Cloud SQL service given new mysql requirements for 1.10. > > When executing "airflow upgradedb" > ``` > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 32, in > args.func(args) > File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, > in initdb > db_utils.initdb(settings.RBAC) > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, > in initdb > upgradedb() > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, > in upgradedb > command.upgrade(config, 'heads') > File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, > in upgrade > script.run_env() > File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line > 416, in run_env > util.load_python_file(self.dir, 'env.py') > File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line > 93, in load_python_file > module = load_module_py(module_id, path) > File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line > 68, in load_module_py > module_id, path).load_module(module_id) > File "", line 399, in > _check_name_wrapper > File "", line 823, in load_module > File "", line 682, in load_module > File "", line 265, in _load_module_shim > File "", line 684, in _load > File "", line 665, in _load_unlocked > File "", line 678, in exec_module > File "", line 219, in _call_with_frames_removed > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 91, in > run_migrations_online() > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 86, in run_migrations_online > context.run_migrations() > File "", line 8, in run_migrations > File > "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line > 807, in run_migrations > self.get_context().run_migrations(**kw) > File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", > line 321, in run_migrations > step.migration_fn(**kw) > File > "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", > line 46, in upgrade > raise Exception("Global variable explicit_defaults_for_timestamp needs to be > on (1) for mysql") > Exception: Global variable explicit_defaults_for_timestamp needs to be on > (1) for mysql > ``` > > Reading documentation for upgrading to airflow 1.10, it seems the requirement > for explicit_defaults_for_timestamp=1 was intentional. > > However, MySQL on Google Cloud SQL does not support configuring this > variable and it is off by default. Users of MySQL and Cloud SQL do not have > an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor. > In GCP, Cloud SQL is _the_ hosted MySQL solution. > [https://cloud.google.com/sql/docs/mysql/flags] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3122) Logs not being printed to STDOUT when running `airflow test` in airflow >= 1.10
[ https://issues.apache.org/jira/browse/AIRFLOW-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fred Israel updated AIRFLOW-3122: - Summary: Logs not being printed to STDOUT when running `airflow test` in airflow >= 1.10 (was: Logs not being printed to STDOUT when running `airflow test` in airflow > 1.9) > Logs not being printed to STDOUT when running `airflow test` in airflow >= > 1.10 > --- > > Key: AIRFLOW-3122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3122 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.0 > Environment: Linux >Reporter: Fred Israel >Priority: Major > Attachments: airflow_bug.zip > > > > In Version 1.10 it seems like there is a regression, as logs in bash operator > are not being sent to stdout in `airflow test` command. This looks wrong to > me, as it makes the airflow test command a bit useless and impossible to > debug. Take a look at my tests below: > (code is in attachment, inside docker for easy testing) > dag.py > > {code:java} > from datetime import datetime > from airflow import DAG > from airflow.operators.bash_operator import BashOperator > dag = DAG('test') > start = datetime.now() > with dag: > BashOperator(task_id='test', bash_command='echo TEST', start_date=start) > BashOperator(task_id='test_failed', bash_command='echo TEST ; exit 1', > start_date=start) > {code} > > > $ VERSION=1.10.0 docker-compose run --rm bug airflow test test test 2000-1-1 > {code:java} > [2018-09-26 20:23:21,614] {__init__.py:51} INFO - Using executor > SequentialExecutor > [2018-09-26 20:23:21,756] {models.py:258} INFO - Filling up the DagBag from > /root/airflow/dags > [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:54} WARNING - Could > not import KubernetesPodOperator: No module named 'kubernetes' > [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:55} WARNING - > Install kubernetes dependencies with: pip install airflow['kubernetes']{code} > $ VERSION=1.9.0 docker-compose run --rm bug airflow test test test 2000-1-1 > {code:java} > [2018-09-26 20:23:28,941] {__init__.py:45} INFO - Using executor > SequentialExecutor > [2018-09-26 20:23:28,994] {models.py:189} INFO - Filling up the DagBag from > /root/airflow/dags > [2018-09-26 20:23:29,094] {bash_operator.py:70} INFO - Tmp dir root location: > /tmp > [2018-09-26 20:23:29,094] {bash_operator.py:80} INFO - Temporary script > location: /tmp/airflowtmphywv1usk//tmp/airflowtmphywv1usk/testoxl3mg4r > [2018-09-26 20:23:29,095] {bash_operator.py:88} INFO - Running command: echo > TEST > [2018-09-26 20:23:29,098] {bash_operator.py:97} INFO - Output: > [2018-09-26 20:23:29,099] {bash_operator.py:101} INFO - TEST > [2018-09-26 20:23:29,099] {bash_operator.py:105} INFO - Command exited with > return code 0{code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3122) Logs not being printed to STDOUT when running `airflow test` in airflow > 1.9
Fred Israel created AIRFLOW-3122: Summary: Logs not being printed to STDOUT when running `airflow test` in airflow > 1.9 Key: AIRFLOW-3122 URL: https://issues.apache.org/jira/browse/AIRFLOW-3122 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.0 Environment: Linux Reporter: Fred Israel Attachments: airflow_bug.zip In Version 1.10 it seems like there is a regression, as logs in bash operator are not being sent to stdout in `airflow test` command. This looks wrong to me, as it makes the airflow test command a bit useless and impossible to debug. Take a look at my tests below: (code is in attachment, inside docker for easy testing) dag.py {code:java} from datetime import datetime from airflow import DAG from airflow.operators.bash_operator import BashOperator dag = DAG('test') start = datetime.now() with dag: BashOperator(task_id='test', bash_command='echo TEST', start_date=start) BashOperator(task_id='test_failed', bash_command='echo TEST ; exit 1', start_date=start) {code} $ VERSION=1.10.0 docker-compose run --rm bug airflow test test test 2000-1-1 {code:java} [2018-09-26 20:23:21,614] {__init__.py:51} INFO - Using executor SequentialExecutor [2018-09-26 20:23:21,756] {models.py:258} INFO - Filling up the DagBag from /root/airflow/dags [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:54} WARNING - Could not import KubernetesPodOperator: No module named 'kubernetes' [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:55} WARNING - Install kubernetes dependencies with: pip install airflow['kubernetes']{code} $ VERSION=1.9.0 docker-compose run --rm bug airflow test test test 2000-1-1 {code:java} [2018-09-26 20:23:28,941] {__init__.py:45} INFO - Using executor SequentialExecutor [2018-09-26 20:23:28,994] {models.py:189} INFO - Filling up the DagBag from /root/airflow/dags [2018-09-26 20:23:29,094] {bash_operator.py:70} INFO - Tmp dir root location: /tmp [2018-09-26 20:23:29,094] {bash_operator.py:80} INFO - Temporary script location: /tmp/airflowtmphywv1usk//tmp/airflowtmphywv1usk/testoxl3mg4r [2018-09-26 20:23:29,095] {bash_operator.py:88} INFO - Running command: echo TEST [2018-09-26 20:23:29,098] {bash_operator.py:97} INFO - Output: [2018-09-26 20:23:29,099] {bash_operator.py:101} INFO - TEST [2018-09-26 20:23:29,099] {bash_operator.py:105} INFO - Command exited with return code 0{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[ https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629341#comment-16629341 ] Iuliia Volkova commented on AIRFLOW-3036: - [~smith-m] please set task unassigned > Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL > > > Key: AIRFLOW-3036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3036 > Project: Apache Airflow > Issue Type: Bug > Components: core, db >Affects Versions: 1.10.0 > Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow > 1.10 on Debian Stretch, Google Cloud SQL MySQL >Reporter: Smith Mathieu >Assignee: Iuliia Volkova >Priority: Blocker > Labels: 1.10, google, google-cloud-sql > Fix For: 2.0.0 > > > The upgrade path to airflow 1.10 seems impossible for users of MySQL in > Google's Cloud SQL service given new mysql requirements for 1.10. > > When executing "airflow upgradedb" > ``` > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 32, in > args.func(args) > File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, > in initdb > db_utils.initdb(settings.RBAC) > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, > in initdb > upgradedb() > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, > in upgradedb > command.upgrade(config, 'heads') > File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, > in upgrade > script.run_env() > File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line > 416, in run_env > util.load_python_file(self.dir, 'env.py') > File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line > 93, in load_python_file > module = load_module_py(module_id, path) > File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line > 68, in load_module_py > module_id, path).load_module(module_id) > File "", line 399, in > _check_name_wrapper > File "", line 823, in load_module > File "", line 682, in load_module > File "", line 265, in _load_module_shim > File "", line 684, in _load > File "", line 665, in _load_unlocked > File "", line 678, in exec_module > File "", line 219, in _call_with_frames_removed > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 91, in > run_migrations_online() > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 86, in run_migrations_online > context.run_migrations() > File "", line 8, in run_migrations > File > "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line > 807, in run_migrations > self.get_context().run_migrations(**kw) > File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", > line 321, in run_migrations > step.migration_fn(**kw) > File > "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", > line 46, in upgrade > raise Exception("Global variable explicit_defaults_for_timestamp needs to be > on (1) for mysql") > Exception: Global variable explicit_defaults_for_timestamp needs to be on > (1) for mysql > ``` > > Reading documentation for upgrading to airflow 1.10, it seems the requirement > for explicit_defaults_for_timestamp=1 was intentional. > > However, MySQL on Google Cloud SQL does not support configuring this > variable and it is off by default. Users of MySQL and Cloud SQL do not have > an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor. > In GCP, Cloud SQL is _the_ hosted MySQL solution. > [https://cloud.google.com/sql/docs/mysql/flags] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[ https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629339#comment-16629339 ] Iuliia Volkova commented on AIRFLOW-3036: - [~smith-m] I'm not sure what somebody could resolve it without Bolke [~bolke] > Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL > > > Key: AIRFLOW-3036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3036 > Project: Apache Airflow > Issue Type: Bug > Components: core, db >Affects Versions: 1.10.0 > Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow > 1.10 on Debian Stretch, Google Cloud SQL MySQL >Reporter: Smith Mathieu >Assignee: Iuliia Volkova >Priority: Blocker > Labels: 1.10, google, google-cloud-sql > Fix For: 2.0.0 > > > The upgrade path to airflow 1.10 seems impossible for users of MySQL in > Google's Cloud SQL service given new mysql requirements for 1.10. > > When executing "airflow upgradedb" > ``` > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 32, in > args.func(args) > File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, > in initdb > db_utils.initdb(settings.RBAC) > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, > in initdb > upgradedb() > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, > in upgradedb > command.upgrade(config, 'heads') > File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, > in upgrade > script.run_env() > File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line > 416, in run_env > util.load_python_file(self.dir, 'env.py') > File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line > 93, in load_python_file > module = load_module_py(module_id, path) > File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line > 68, in load_module_py > module_id, path).load_module(module_id) > File "", line 399, in > _check_name_wrapper > File "", line 823, in load_module > File "", line 682, in load_module > File "", line 265, in _load_module_shim > File "", line 684, in _load > File "", line 665, in _load_unlocked > File "", line 678, in exec_module > File "", line 219, in _call_with_frames_removed > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 91, in > run_migrations_online() > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 86, in run_migrations_online > context.run_migrations() > File "", line 8, in run_migrations > File > "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line > 807, in run_migrations > self.get_context().run_migrations(**kw) > File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", > line 321, in run_migrations > step.migration_fn(**kw) > File > "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", > line 46, in upgrade > raise Exception("Global variable explicit_defaults_for_timestamp needs to be > on (1) for mysql") > Exception: Global variable explicit_defaults_for_timestamp needs to be on > (1) for mysql > ``` > > Reading documentation for upgrading to airflow 1.10, it seems the requirement > for explicit_defaults_for_timestamp=1 was intentional. > > However, MySQL on Google Cloud SQL does not support configuring this > variable and it is off by default. Users of MySQL and Cloud SQL do not have > an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor. > In GCP, Cloud SQL is _the_ hosted MySQL solution. > [https://cloud.google.com/sql/docs/mysql/flags] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yeluolei commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
yeluolei commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-424853136 @odracci please merge change #3770 first, I will try to rebase this commit if needed. if you can add this feature in your change, it will be better. thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yeluolei commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
yeluolei commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#discussion_r220705601 ## File path: airflow/contrib/kubernetes/worker_configuration.py ## @@ -121,32 +121,19 @@ def _construct_volume(name, claim): return volume volumes = [ -_construct_volume( -dags_volume_name, -self.kube_config.dags_volume_claim -), _construct_volume( logs_volume_name, self.kube_config.logs_volume_claim ) ] -dag_volume_mount_path = "" - -if self.kube_config.dags_volume_claim: -dag_volume_mount_path = self.worker_airflow_dags -else: -dag_volume_mount_path = os.path.join( -self.worker_airflow_dags, -self.kube_config.git_subpath +if not self.kube_config.dags_in_docker: +volumes.append( +_construct_volume( +dags_volume_name, +self.kube_config.dags_volume_claim +) ) -dags_volume_mount = { -'name': dags_volume_name, -'mountPath': dag_volume_mount_path, -'readOnly': True, -} -if self.kube_config.dags_volume_subpath: -dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath logs_volume_mount = { Review comment: I agree with this, actually I need this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aoen commented on issue #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter
aoen commented on issue #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter URL: https://github.com/apache/incubator-airflow/pull/3955#issuecomment-424848684 LGTM once CI passes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3121) StreamLogWriter should extend io.IOBase
[ https://issues.apache.org/jira/browse/AIRFLOW-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629309#comment-16629309 ] ASF GitHub Bot commented on AIRFLOW-3121: - jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed property on LoggingMixin URL: https://github.com/apache/incubator-airflow/pull/3955 ### Jira - [X] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3121 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: This PR adds the `closed` property to the `StreamLogWriter` class—a stub function that simply returns `False` for compatibility reasons. ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: This PR adds a straightforward property. ### Commits - [X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [X] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > StreamLogWriter should extend io.IOBase > --- > > Key: AIRFLOW-3121 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3121 > Project: Apache Airflow > Issue Type: Bug >Reporter: Jonathan >Assignee: Jonathan >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed property on LoggingMixin
jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed property on LoggingMixin URL: https://github.com/apache/incubator-airflow/pull/3955 ### Jira - [X] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3121 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: This PR adds the `closed` property to the `StreamLogWriter` class—a stub function that simply returns `False` for compatibility reasons. ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: This PR adds a straightforward property. ### Commits - [X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [X] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3121) StreamLogWriter should extend io.IOBase
Jonathan created AIRFLOW-3121: - Summary: StreamLogWriter should extend io.IOBase Key: AIRFLOW-3121 URL: https://issues.apache.org/jira/browse/AIRFLOW-3121 Project: Apache Airflow Issue Type: Bug Reporter: Jonathan Assignee: Jonathan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jakahn commented on issue #3805: [AIRFLOW-2062] Add per-connection KMS encryption.
jakahn commented on issue #3805: [AIRFLOW-2062] Add per-connection KMS encryption. URL: https://github.com/apache/incubator-airflow/pull/3805#issuecomment-424837465 @Fokko No problem! Rebased and pushed, Travis running now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dimberman commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
dimberman commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#discussion_r220668743 ## File path: airflow/contrib/kubernetes/worker_configuration.py ## @@ -121,32 +121,19 @@ def _construct_volume(name, claim): return volume volumes = [ -_construct_volume( -dags_volume_name, -self.kube_config.dags_volume_claim -), _construct_volume( logs_volume_name, self.kube_config.logs_volume_claim ) ] -dag_volume_mount_path = "" - -if self.kube_config.dags_volume_claim: -dag_volume_mount_path = self.worker_airflow_dags -else: -dag_volume_mount_path = os.path.join( -self.worker_airflow_dags, -self.kube_config.git_subpath +if not self.kube_config.dags_in_docker: +volumes.append( +_construct_volume( +dags_volume_name, +self.kube_config.dags_volume_claim +) ) -dags_volume_mount = { -'name': dags_volume_name, -'mountPath': dag_volume_mount_path, -'readOnly': True, -} -if self.kube_config.dags_volume_subpath: -dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath logs_volume_mount = { Review comment: Could we please modify this s.t. users can determine not to use a volume for logs? Would be great to completely separate the necessity of volumes for launching. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[ https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629173#comment-16629173 ] Brad Holmes commented on AIRFLOW-3079: -- OK, I already have the fixes locally -- I just need to create a PR, and as this is my first one, it will take me a little while to find the time and figure out how to do it. > initdb fails on Microsoft SQL Server > > > Key: AIRFLOW-3079 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3079 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 1.10.0 >Reporter: Morten Post >Priority: Major > > airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not > exist in 1.9.0. > [*@ airflow]$ airflow initdb > [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm(): > Using pool settings. pool_size=5, pool_recycle=1800 > [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor > SequentialExecutor > DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL > Server > [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables > INFO [alembic.runtime.migration] Context impl MSSQLImpl. > INFO [alembic.runtime.migration] Will assume transactional DDL. > INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current > schema > INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, > create is_encrypted > INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> > 13eb55f81627, maintain history for compatibility with earlier migrations > INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> > 338e90f54d61, More logging into task_isntance > INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, > job_id indices > INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, > Adding extra to Log > INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> > 1b38cef5b76e, add dagrun > INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> > 2e541a1dcfed, task_duration > INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> > 40e67319e3a9, dagrun_config > INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> > 561833c1c74b, add password column to user > INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, > dagrun start end > INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, > Add notification_sent column to sla_miss > INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> > bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field > in connection > INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> > 1968acfc09e3, add is_encrypted column to variable table > INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> > 2e82aab8ef20, rename user table > INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> > 211e584da130, add TI state index > INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> > 64de9cddf6c9, add task fails journal table > INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> > f2ca10b85618, add dag_stats table > INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> > 4addfa1236f1, Add fractional seconds to mysql tables > INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> > 8504051e801b, xcom dag task indices > INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> > 5e7d17757c7a, add pid field to TaskInstance > INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> > 127d2bf2dfa7, Add dag_id/state index on dag_run table > INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> > cc1e65623dc7, add max tries column to task instance > INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> > bdaa763e6c56, Make xcom value column a large binary > INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> > 947454bf1dff, add ti job_id index > INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> > d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text > types) > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/bin/airflow", line 32, in > args.func(args) > File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in > initdb > db_utils.initdb(settings.RBAC) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in > initdb > upgradedb() > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in > upgradedb > command.upgrade(config, 'heads') > File
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[ https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629169#comment-16629169 ] Brad Holmes commented on AIRFLOW-3079: -- Technically, this ticket duplicates AIRFLOW-1877, as it came later. But, the activity is here, so we'll say it is duplicated by AIRFLOW-1877, which should probably be closed in favor of this ticket. > initdb fails on Microsoft SQL Server > > > Key: AIRFLOW-3079 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3079 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 1.10.0 >Reporter: Morten Post >Priority: Major > > airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not > exist in 1.9.0. > [*@ airflow]$ airflow initdb > [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm(): > Using pool settings. pool_size=5, pool_recycle=1800 > [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor > SequentialExecutor > DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL > Server > [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables > INFO [alembic.runtime.migration] Context impl MSSQLImpl. > INFO [alembic.runtime.migration] Will assume transactional DDL. > INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current > schema > INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, > create is_encrypted > INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> > 13eb55f81627, maintain history for compatibility with earlier migrations > INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> > 338e90f54d61, More logging into task_isntance > INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, > job_id indices > INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, > Adding extra to Log > INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> > 1b38cef5b76e, add dagrun > INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> > 2e541a1dcfed, task_duration > INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> > 40e67319e3a9, dagrun_config > INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> > 561833c1c74b, add password column to user > INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, > dagrun start end > INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, > Add notification_sent column to sla_miss > INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> > bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field > in connection > INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> > 1968acfc09e3, add is_encrypted column to variable table > INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> > 2e82aab8ef20, rename user table > INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> > 211e584da130, add TI state index > INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> > 64de9cddf6c9, add task fails journal table > INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> > f2ca10b85618, add dag_stats table > INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> > 4addfa1236f1, Add fractional seconds to mysql tables > INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> > 8504051e801b, xcom dag task indices > INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> > 5e7d17757c7a, add pid field to TaskInstance > INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> > 127d2bf2dfa7, Add dag_id/state index on dag_run table > INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> > cc1e65623dc7, add max tries column to task instance > INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> > bdaa763e6c56, Make xcom value column a large binary > INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> > 947454bf1dff, add ti job_id index > INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> > d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text > types) > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/bin/airflow", line 32, in > args.func(args) > File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in > initdb > db_utils.initdb(settings.RBAC) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in > initdb > upgradedb() > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in > upgradedb > command.upgrade(config, 'heads') > File
[jira] [Reopened] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[ https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Smith Mathieu reopened AIRFLOW-3036: Assignee: Iuliia Volkova (was: Josh Carp) The PR that autoclosed this issue was misidentified as being associated with AIRFLOW-3036 (this issue). The PR that closed this issue actually addressed AIRFLOW-3074. This issue has not been addressed > Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL > > > Key: AIRFLOW-3036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3036 > Project: Apache Airflow > Issue Type: Bug > Components: core, db >Affects Versions: 1.10.0 > Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow > 1.10 on Debian Stretch, Google Cloud SQL MySQL >Reporter: Smith Mathieu >Assignee: Iuliia Volkova >Priority: Blocker > Labels: 1.10, google, google-cloud-sql > Fix For: 2.0.0 > > > The upgrade path to airflow 1.10 seems impossible for users of MySQL in > Google's Cloud SQL service given new mysql requirements for 1.10. > > When executing "airflow upgradedb" > ``` > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 32, in > args.func(args) > File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, > in initdb > db_utils.initdb(settings.RBAC) > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, > in initdb > upgradedb() > File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, > in upgradedb > command.upgrade(config, 'heads') > File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, > in upgrade > script.run_env() > File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line > 416, in run_env > util.load_python_file(self.dir, 'env.py') > File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line > 93, in load_python_file > module = load_module_py(module_id, path) > File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line > 68, in load_module_py > module_id, path).load_module(module_id) > File "", line 399, in > _check_name_wrapper > File "", line 823, in load_module > File "", line 682, in load_module > File "", line 265, in _load_module_shim > File "", line 684, in _load > File "", line 665, in _load_unlocked > File "", line 678, in exec_module > File "", line 219, in _call_with_frames_removed > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 91, in > run_migrations_online() > File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", > line 86, in run_migrations_online > context.run_migrations() > File "", line 8, in run_migrations > File > "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line > 807, in run_migrations > self.get_context().run_migrations(**kw) > File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", > line 321, in run_migrations > step.migration_fn(**kw) > File > "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", > line 46, in upgrade > raise Exception("Global variable explicit_defaults_for_timestamp needs to be > on (1) for mysql") > Exception: Global variable explicit_defaults_for_timestamp needs to be on > (1) for mysql > ``` > > Reading documentation for upgrading to airflow 1.10, it seems the requirement > for explicit_defaults_for_timestamp=1 was intentional. > > However, MySQL on Google Cloud SQL does not support configuring this > variable and it is off by default. Users of MySQL and Cloud SQL do not have > an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor. > In GCP, Cloud SQL is _the_ hosted MySQL solution. > [https://cloud.google.com/sql/docs/mysql/flags] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] smith-m commented on issue #3908: [AIRFLOW-3036] Add relevant ECS options to ECS operator.
smith-m commented on issue #3908: [AIRFLOW-3036] Add relevant ECS options to ECS operator. URL: https://github.com/apache/incubator-airflow/pull/3908#issuecomment-424795372 In that case, the airflow-3036 needs to be reopened and 3074 should be closed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 commented on issue #3954: Update README.md
r39132 commented on issue #3954: Update README.md URL: https://github.com/apache/incubator-airflow/pull/3954#issuecomment-424793393 Closing dupe of https://github.com/apache/incubator-airflow/pull/3953 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 opened a new pull request #3954: Update README.md
r39132 opened a new pull request #3954: Update README.md URL: https://github.com/apache/incubator-airflow/pull/3954 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 closed pull request #3954: Update README.md
r39132 closed pull request #3954: Update README.md URL: https://github.com/apache/incubator-airflow/pull/3954 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 closed pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme
r39132 closed pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme URL: https://github.com/apache/incubator-airflow/pull/3953 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 opened a new pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme
r39132 opened a new pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme URL: https://github.com/apache/incubator-airflow/pull/3953 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 closed pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists
r39132 closed pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists URL: https://github.com/apache/incubator-airflow/pull/3951 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp commented on issue #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py.
jmcarp commented on issue #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py. URL: https://github.com/apache/incubator-airflow/pull/3917#issuecomment-424785093 Does this make sense @Fokko? Should be quick to review when somebody has time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akshayi1 opened a new pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code
akshayi1 opened a new pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code URL: https://github.com/apache/incubator-airflow/pull/3952 Updated documentation for the steps to set up remote logging to Google Cloud Storage. In discussion with @ashb on the Slack channel, this does not need a JIRA ticket. Please take a look and approve this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-2794) Add delete support for Azure blob
[ https://issues.apache.org/jira/browse/AIRFLOW-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-2794: -- Assignee: Bart Eijk > Add delete support for Azure blob > - > > Key: AIRFLOW-2794 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2794 > Project: Apache Airflow > Issue Type: Wish > Components: hooks, operators >Reporter: Bart Eijk >Assignee: Bart Eijk >Priority: Trivial > > As a developer, I would like to have the ability to create tasks that can > delete files in Azure blob storage. > Nice to have: the ability to delete a "folder", i.e. a prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3120) Missing dependency for gcp_api authentification
Victor created AIRFLOW-3120: --- Summary: Missing dependency for gcp_api authentification Key: AIRFLOW-3120 URL: https://issues.apache.org/jira/browse/AIRFLOW-3120 Project: Apache Airflow Issue Type: Bug Components: authentication Affects Versions: 1.10.0 Reporter: Victor In order to use Google Auth with Airflow, the package Flask-OAuthlib is needed. I was expecting that passing the option gcp_api during pip install would add the dependency. Another solution would be to add a new option google_auth or something like that of course. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] r39132 opened a new pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists
r39132 opened a new pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists URL: https://github.com/apache/incubator-airflow/pull/3951 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ckljohn commented on issue #3945: [AIRFLOW-3112] Make SFTP hook to inherit SSH hook
ckljohn commented on issue #3945: [AIRFLOW-3112] Make SFTP hook to inherit SSH hook URL: https://github.com/apache/incubator-airflow/pull/3945#issuecomment-424740292 Should the backward compatibility be deprecated on the next major release? Should I also include a deprecate note? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[ https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628834#comment-16628834 ] Ash Berlin-Taylor commented on AIRFLOW-3079: Ah, "thanks SQLServer!" Probably fixing the migration in question - it is already specialised to cope with different DB engines, so extending it to cope with SQLServer is probably not a lot of work. > initdb fails on Microsoft SQL Server > > > Key: AIRFLOW-3079 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3079 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 1.10.0 >Reporter: Morten Post >Priority: Major > > airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not > exist in 1.9.0. > [*@ airflow]$ airflow initdb > [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm(): > Using pool settings. pool_size=5, pool_recycle=1800 > [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor > SequentialExecutor > DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL > Server > [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables > INFO [alembic.runtime.migration] Context impl MSSQLImpl. > INFO [alembic.runtime.migration] Will assume transactional DDL. > INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current > schema > INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, > create is_encrypted > INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> > 13eb55f81627, maintain history for compatibility with earlier migrations > INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> > 338e90f54d61, More logging into task_isntance > INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, > job_id indices > INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, > Adding extra to Log > INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> > 1b38cef5b76e, add dagrun > INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> > 2e541a1dcfed, task_duration > INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> > 40e67319e3a9, dagrun_config > INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> > 561833c1c74b, add password column to user > INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, > dagrun start end > INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, > Add notification_sent column to sla_miss > INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> > bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field > in connection > INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> > 1968acfc09e3, add is_encrypted column to variable table > INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> > 2e82aab8ef20, rename user table > INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> > 211e584da130, add TI state index > INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> > 64de9cddf6c9, add task fails journal table > INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> > f2ca10b85618, add dag_stats table > INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> > 4addfa1236f1, Add fractional seconds to mysql tables > INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> > 8504051e801b, xcom dag task indices > INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> > 5e7d17757c7a, add pid field to TaskInstance > INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> > 127d2bf2dfa7, Add dag_id/state index on dag_run table > INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> > cc1e65623dc7, add max tries column to task instance > INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> > bdaa763e6c56, Make xcom value column a large binary > INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> > 947454bf1dff, add ti job_id index > INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> > d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text > types) > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add time zone awareness > Traceback (most recent call last): > File "/bin/airflow", line 32, in > args.func(args) > File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in > initdb > db_utils.initdb(settings.RBAC) > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in > initdb > upgradedb() > File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in > upgradedb > command.upgrade(config, 'heads') > File
[jira] [Commented] (AIRFLOW-3119) Enable loglevel on celery worker and inherit from airflow.cfg
[ https://issues.apache.org/jira/browse/AIRFLOW-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628831#comment-16628831 ] ASF GitHub Bot commented on AIRFLOW-3119: - cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg URL: https://github.com/apache/incubator-airflow/pull/3950 Make sure you have checked _all_ steps below. ### Jira - [x ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ x] Enable debugging on my celery worker, this story will enable --loglevel when launching a celery worker: http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html and inherit that loglevel setting from airflow.cfg ### Tests - [x ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Enable loglevel on celery worker and inherit from airflow.cfg > - > > Key: AIRFLOW-3119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3119 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, worker >Affects Versions: 1.9.0, 1.10.0 >Reporter: Luke Bodeen >Assignee: Luke Bodeen >Priority: Minor > Fix For: 1.10.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > I would like to enable debugging on my celery worker, this story will enable > --loglevel when launching a celery worker: > [http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html] > > and inherit that loglevel setting from airflow.cfg -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg
cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg URL: https://github.com/apache/incubator-airflow/pull/3950 Make sure you have checked _all_ steps below. ### Jira - [x ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description - [ x] Enable debugging on my celery worker, this story will enable --loglevel when launching a celery worker: http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html and inherit that loglevel setting from airflow.cfg ### Tests - [x ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [x ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x ] In case of new functionality, my PR adds documentation that describes how to use it. - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added. ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3119) Enable loglevel on celery worker and inherit from airflow.cfg
Luke Bodeen created AIRFLOW-3119: Summary: Enable loglevel on celery worker and inherit from airflow.cfg Key: AIRFLOW-3119 URL: https://issues.apache.org/jira/browse/AIRFLOW-3119 Project: Apache Airflow Issue Type: Improvement Components: celery, worker Affects Versions: 1.10.0, 1.9.0 Reporter: Luke Bodeen Assignee: Luke Bodeen Fix For: 1.10.1 I would like to enable debugging on my celery worker, this story will enable --loglevel when launching a celery worker: [http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html] and inherit that loglevel setting from airflow.cfg -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[ https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628785#comment-16628785 ] Brad Holmes commented on AIRFLOW-3079: -- I also encountered this problem. The root cause is that {{TIMESTAMP}} for Microsoft SQL Server ({{mssql}}) is actually a rowid, where as for others, it is what it sounds like: a timestamp. See https://stackoverflow.com/questions/41682748/sqlalchemy-fails-to-insert-timestamp-into-mssql, especially this comment: {quote} Microsoft SQL Server's timestamp data type is not, actually, a timestamp - it is more of a sequence (it is an ancient holdover, and horribly misnamed). If you need to update a date time value, use a datetime data type. {quote} I am unsure if we should apply the fix here in Airflow (I have a patch that works) or try to get a fix into {{sqlalchemy}}. > initdb fails on Microsoft SQL Server > > > Key: AIRFLOW-3079 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3079 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 1.10.0 >Reporter: Morten Post >Priority: Major > > airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not > exist in 1.9.0. > [*@ airflow]$ airflow initdb > [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm(): > Using pool settings. pool_size=5, pool_recycle=1800 > [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor > SequentialExecutor > DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL > Server > [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables > INFO [alembic.runtime.migration] Context impl MSSQLImpl. > INFO [alembic.runtime.migration] Will assume transactional DDL. > INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current > schema > INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, > create is_encrypted > INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> > 13eb55f81627, maintain history for compatibility with earlier migrations > INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> > 338e90f54d61, More logging into task_isntance > INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, > job_id indices > INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, > Adding extra to Log > INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> > 1b38cef5b76e, add dagrun > INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> > 2e541a1dcfed, task_duration > INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> > 40e67319e3a9, dagrun_config > INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> > 561833c1c74b, add password column to user > INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, > dagrun start end > INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, > Add notification_sent column to sla_miss > INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> > bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field > in connection > INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> > 1968acfc09e3, add is_encrypted column to variable table > INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> > 2e82aab8ef20, rename user table > INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> > 211e584da130, add TI state index > INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> > 64de9cddf6c9, add task fails journal table > INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> > f2ca10b85618, add dag_stats table > INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> > 4addfa1236f1, Add fractional seconds to mysql tables > INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> > 8504051e801b, xcom dag task indices > INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> > 5e7d17757c7a, add pid field to TaskInstance > INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> > 127d2bf2dfa7, Add dag_id/state index on dag_run table > INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> > cc1e65623dc7, add max tries column to task instance > INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> > bdaa763e6c56, Make xcom value column a large binary > INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> > 947454bf1dff, add ti job_id index > INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> > d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text > types) > INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> > 0e2a74e0fc9f, Add
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628764#comment-16628764 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- Here is a screenshot of the example_bash_operator DAG logs table: !Screenshot_20180926_161837.png! > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: Screenshot_20180926_161837.png, > image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628750#comment-16628750 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- The start_date for the example_bash_operator DAG is 'two days ago': {code:java} args = { 'owner': 'airflow', 'start_date': airflow.utils.dates.days_ago(2) } {code} > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628761#comment-16628761 ] Ash Berlin-Taylor commented on AIRFLOW-3118: If no logs appear is the scheduler still running? Alternatively do you get any output/error messages when running {{airflow scheduler}}? They might be rather verbose and repeating, so only the first few (100? 50?) lines are all that interesting probably. > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brylie Christopher Oxley updated AIRFLOW-3118: -- Attachment: Screenshot_20180926_161837.png > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: Screenshot_20180926_161837.png, > image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628753#comment-16628753 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- Although the logs/scheduler directory is created, with two sub-directories (latest and 2018-09-26), no logs appear. > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628740#comment-16628740 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- The example_bash_operator DAG looks like this: {code:java} dag = DAG( dag_id='example_bash_operator', default_args=args, schedule_interval='0 0 * * *', dagrun_timeout=timedelta(minutes=60)) {code} It seems to be scheduled on an interval, rather than at a date. > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628735#comment-16628735 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- Here is the output from airlow list_dags -r {noformat} --- DAGS --- example_bash_operator example_branch_dop_operator_v3 example_branch_operator example_http_operator example_kubernetes_executor example_passing_params_via_test_command example_python_operator example_short_circuit_operator example_skip_dag example_subdag_operator example_subdag_operator.section-1 example_subdag_operator.section-2 example_trigger_controller_dag example_trigger_target_dag example_xcom latest_only latest_only_with_trigger test_utils tutorial --- DagBag loading stats for /home/brylie/airflow/dags --- Number of DAGs: 0 Total task number: 0 DagBag parsing time: 0 None {noformat} I am trying to run any of the example DAGs that ship with Airflow. > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-424696610 Please add more docs, possibly including the diagram - having more detail about how the scheduler works in our docs would be a Very Good Thing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body
ron819 commented on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body URL: https://github.com/apache/incubator-airflow/pull/2334#issuecomment-424689055 @Fokko any news? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2015: [AIRFLOW-765] Auto detect dag dependency files, variables, and resour…
ron819 commented on issue #2015: [AIRFLOW-765] Auto detect dag dependency files, variables, and resour… URL: https://github.com/apache/incubator-airflow/pull/2015#issuecomment-424687630 @artwr @aminghadersohi is this still active? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mishikaSingh commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash
mishikaSingh commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash URL: https://github.com/apache/incubator-airflow/pull/3650#issuecomment-424680530 @ashb As we are handling only OperationalError so this will handle only "Exception raised for errors that are related to the database's operation and not necessarily under the control of the programmer, e.g. an unexpected disconnect occurs, the data source name is not found, a transaction could not be processed, a memory allocation error occurred during processing, etc." we think that these intermittent issues should not bring the whole scheduler down. Any other DB related issue will bring the scheduler down as they will come under different class . |__Error |__InterfaceError |__DatabaseError |__DataError |___OperationalError_ |__IntegrityError |__InternalError |__ProgrammingError |__NotSupportedError This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] BasPH commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
BasPH commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424671583 Thanks. Will see if I can process @Fokko 's comments tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628555#comment-16628555 ] Ash Berlin-Taylor commented on AIRFLOW-3118: Can you check in the scheduler log? See if there is anything "interesting" there (or attach a small sample there). One other thing to check: In the terminal where you will run the scheduler run {{airflow list_dags -r}} - it should list your example_bash_operator. Does it? What start date have you set on the dag? > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628551#comment-16628551 ] Brylie Christopher Oxley commented on AIRFLOW-3118: --- We have, so far, tried to set up Airflow on two computers (Linux and Mac). In both cases, the example DAGs never get out of the 'running' state. > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brylie Christopher Oxley updated AIRFLOW-3118: -- Description: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** conda create -n airflow ** source activate airflow # install airflow ** pip install apache-airflow # initialize Airflow db ** airflow initdb # disable default paused setting in airflow.cfg ** dags_are_paused_at_creation = False # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}airflow scheduler{color} ** {color:#6a8759}airflow webserver{color} # {color:#6a8759}unpause example_bash_operator{color} ** {color:#6a8759}airflow unpause example_bash_operator{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on example_bash_operator{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} was: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** conda create -n airflow ** source activate airflow # install airflow ** pip install apache-airflow # initialize Airflow db ** airflow initdb # disable default paused setting in airflow.cfg ** dags_are_paused_at_creation = False # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}airflow scheduler{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause example_bash_operator{color} ** {color:#6a8759}airflow unpause example_bash_operator{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on example_bash_operator{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}airflow webserver{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3106) Validate Postgres connection when saving
[ https://issues.apache.org/jira/browse/AIRFLOW-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-3106: --- Priority: Minor (was: Major) Summary: Validate Postgres connection when saving (was: Validate connection when saving connection) > Validate Postgres connection when saving > > > Key: AIRFLOW-3106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3106 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Priority: Minor > > I've encountered failures in DAG runs at various occasions due to invalid > connection credentials, or a domain was unreachable from the Airflow instance. > It'd be nice to validate a connection when saving it, to directly know if a > given connection can be made or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
ashb commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424664381 Oh yes, I was basing my comment on the title of the this Jira - I've changed that to be more specific and linked the two issues. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3111) Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
[ https://issues.apache.org/jira/browse/AIRFLOW-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-3111: --- Fix Version/s: 1.10.1 > Confusing comments and instructions for log templates in UPDATING.md and > default_airflow.cfg > > > Key: AIRFLOW-3111 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3111 > Project: Apache Airflow > Issue Type: Bug >Reporter: Eric Chang >Assignee: Eric Chang >Priority: Minor > Fix For: 1.10.1 > > > The new 1.10 release includes instructions on updating the *airflow.cfg* > options core.log_filename_template and > elasticsearch.elasticsearch_log_id_template that are technically incorrect: > > {code:java} > // UPDATING.md > ...the following defaults need to be added. > [core] > log_filename_template = ti.dag_id / ti.task_id / ts > / try_number .log > [elasticsearch] > elasticsearch_log_id_template = > {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}{code} > > Inserting the above options into an existing *airflow.cfg* will result in a > jinja parse failure because `` is not a valid template string. The reason > the extra braces are necessary is because the default *airflow.cfg* copied to > AIRFLOW_HOME is a rendered version of *default_airflow.cfg* (rendered by > *airflow.configuration.parameterized_config*). > The confusion is compounded by several comments copied over to *airflow.cfg* > stating that > > {code:java} > # we need to escape the curly braces by adding an additional curly brace > {code} > But that's is only true for *default_airflow.cfg* and doesn't apply to the > actual *airflow.cfg*. > I'll submit a PR that corrects this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[ https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3109. Resolution: Fixed Fix Version/s: 1.10.1 > Default user permission should contain 'can_clear' > -- > > Key: AIRFLOW-3109 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3109 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > Fix For: 1.10.1 > > > The default user role is missing 'can_clear' permission which allows user to > clear DAG runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641 ## File path: airflow/contrib/hooks/bigquery_hook.py ## @@ -1485,3 +1490,17 @@ def var_print(var_name): project_id = default_project_id return project_id, dataset_id, table_id + + +def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in): +# if it is a partitioned table ($ is in the table name) add partition load option +time_partitioning_out = {} +if destination_dataset_table and '$' in destination_dataset_table: +assert not time_partitioning_in.get('field'), ( Review comment: Sorry, silly me. I meant this will be available in **Airflow 1.10.1** (Next Release) . :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641 ## File path: airflow/contrib/hooks/bigquery_hook.py ## @@ -1485,3 +1490,17 @@ def var_print(var_name): project_id = default_project_id return project_id, dataset_id, table_id + + +def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in): +# if it is a partitioned table ($ is in the table name) add partition load option +time_partitioning_out = {} +if destination_dataset_table and '$' in destination_dataset_table: +assert not time_partitioning_in.get('field'), ( Review comment: Sorry, silly me. I meant this will be available in **Airflow 1.10.1** . :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641 ## File path: airflow/contrib/hooks/bigquery_hook.py ## @@ -1485,3 +1490,17 @@ def var_print(var_name): project_id = default_project_id return project_id, dataset_id, table_id + + +def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in): +# if it is a partitioned table ($ is in the table name) add partition load option +time_partitioning_out = {} +if destination_dataset_table and '$' in destination_dataset_table: +assert not time_partitioning_in.get('field'), ( Review comment: Sorry, silly me. I meant this will be available in *Airflow 1.10.1*. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220342170 ## File path: airflow/contrib/hooks/bigquery_hook.py ## @@ -1485,3 +1490,17 @@ def var_print(var_name): project_id = default_project_id return project_id, dataset_id, table_id + + +def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in): +# if it is a partitioned table ($ is in the table name) add partition load option +time_partitioning_out = {} +if destination_dataset_table and '$' in destination_dataset_table: +assert not time_partitioning_in.get('field'), ( Review comment: This was fixed in https://github.com/apache/incubator-airflow/pull/3901 and will be available in **Airflow 1.10.1** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brylie Christopher Oxley updated AIRFLOW-3118: -- Description: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** conda create -n airflow ** source activate airflow # install airflow ** pip install apache-airflow # initialize Airflow db ** airflow initdb # disable default paused setting in airflow.cfg ** dags_are_paused_at_creation = False # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}airflow scheduler{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause example_bash_operator{color} ** {color:#6a8759}airflow unpause example_bash_operator{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on example_bash_operator{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} was: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** {code:java} conda create -n airflow{code} ** {code:java} source activate airflow{code} # install airflow ** {code:java} pip install apache-airflow{code} # initialize Airflow db ** {code:java} airflow initdb{code} # disable default paused setting in airflow.cfg ## {code:java} dags_are_paused_at_creation = False{code} # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}airflow scheduler{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause `exam`ple_bash_operator`{color} ** {color:#6a8759}`airflow unpause example_bash_operator`{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on `example_bash_operator`{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** conda create -n airflow > ** source activate airflow > # install airflow > ** pip install apache-airflow > # initialize Airflow db > ** airflow initdb > # disable default paused setting in airflow.cfg > ** dags_are_paused_at_creation = False > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}`airflow server`{color} > # {color:#6a8759}unpause example_bash_operator{color} > ** {color:#6a8759}airflow unpause example_bash_operator{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on example_bash_operator{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brylie Christopher Oxley updated AIRFLOW-3118: -- Description: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** {code:java} conda create -n airflow{code} ** {code:java} source activate airflow{code} # install airflow ** {code:java} pip install apache-airflow{code} # initialize Airflow db ** {code:java} airflow initdb{code} # disable default paused setting in airflow.cfg ## {code:java} dags_are_paused_at_creation = False{code} # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}airflow scheduler{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause `exam`ple_bash_operator`{color} ** {color:#6a8759}`airflow unpause example_bash_operator`{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on `example_bash_operator`{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} was: When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** `conda create -n airflow` ** `source activate airflow` # install airflow ** `pip install apache-airflow` # initialize Airflow db ** `airflow initdb` # disable default paused setting in airflow.cfg ## `dags_are_paused_at_creation = {color:#6a8759}False`{color} # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}`airflow scheduler`{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause `exam`ple_bash_operator`{color} ** {color:#6a8759}`airflow unpause example_bash_operator`{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on `example_bash_operator`{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} > DAGs not successful on new installation > --- > > Key: AIRFLOW-3118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.0 > Environment: Ubuntu 18.04 > Python 3.6 >Reporter: Brylie Christopher Oxley >Priority: Blocker > Attachments: image-2018-09-26-12-39-03-094.png > > > When trying out Airflow, on localhost, none of the DAG runs are getting to > the 'success' state. They are getting stuck in 'running', or I manually label > them as failed: > !image-2018-09-26-12-39-03-094.png! > h2. Steps to reproduce > # create new conda environment > ** > {code:java} > conda create -n airflow{code} > ** > {code:java} > source activate airflow{code} > # install airflow > ** > {code:java} > pip install apache-airflow{code} > # initialize Airflow db > ** > {code:java} > airflow initdb{code} > # disable default paused setting in airflow.cfg > ## > {code:java} > dags_are_paused_at_creation = False{code} > # {color:#6a8759}run airflow and airflow scheduler (in separate > terminal){color} > ** {color:#6a8759}airflow scheduler{color} > ** {color:#6a8759}`airflow server`{color} > # {color:#6a8759}unpause `exam`ple_bash_operator`{color} > ** {color:#6a8759}`airflow unpause example_bash_operator`{color} > # {color:#6a8759}log in to Airflow UI{color} > # {color:#6a8759}turn on `example_bash_operator`{color} > # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} > h2. {color:#6a8759}Observed result{color} > {color:#6a8759}The `example_bash_operator` never leaves the "running" > state.{color} > h2. {color:#6a8759}Expected result{color} > {color:#6a8759}The `example_bash_operator` would quickly enter the "success" > state{color} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3118) DAGs not successful on new installation
Brylie Christopher Oxley created AIRFLOW-3118: - Summary: DAGs not successful on new installation Key: AIRFLOW-3118 URL: https://issues.apache.org/jira/browse/AIRFLOW-3118 Project: Apache Airflow Issue Type: Bug Components: DAG Affects Versions: 1.10.0 Environment: Ubuntu 18.04 Python 3.6 Reporter: Brylie Christopher Oxley Attachments: image-2018-09-26-12-39-03-094.png When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed: !image-2018-09-26-12-39-03-094.png! h2. Steps to reproduce # create new conda environment ** `conda create -n airflow` ** `source activate airflow` # install airflow ** `pip install apache-airflow` # initialize Airflow db ** `airflow initdb` # disable default paused setting in airflow.cfg ## `dags_are_paused_at_creation = {color:#6a8759}False`{color} # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color} ** {color:#6a8759}`airflow scheduler`{color} ** {color:#6a8759}`airflow server`{color} # {color:#6a8759}unpause `exam`ple_bash_operator`{color} ** {color:#6a8759}`airflow unpause example_bash_operator`{color} # {color:#6a8759}log in to Airflow UI{color} # {color:#6a8759}turn on `example_bash_operator`{color} # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color} h2. {color:#6a8759}Observed result{color} {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color} h2. {color:#6a8759}Expected result{color} {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2979) Deprecated Celery Option not in Options list
[ https://issues.apache.org/jira/browse/AIRFLOW-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2979: --- Fix Version/s: 1.10.1 > Deprecated Celery Option not in Options list > - > > Key: AIRFLOW-2979 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2979 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.10.0 >Reporter: Micheal Ascah >Assignee: Kaxil Naik >Priority: Critical > Fix For: 2.0.0, 1.10.1 > > > References AIRFLOW-1840 > In airflow/configuration.py > {code:java} > # A two-level mapping of (section -> new_name -> old_name). When reading > # new_name, the old_name will be checked to see if it exists. If it does a > # DeprecationWarning will be issued and the old name will be used instead > deprecated_options = { > 'celery': { > # Remove these keys in Airflow 1.11 > 'worker_concurrency': 'celeryd_concurrency', > 'broker_url': 'celery_broker_url', > 'ssl_active': 'celery_ssl_active', > 'ssl_cert': 'celery_ssl_cert', > 'ssl_key': 'celery_ssl_key', > } > } > {code} > This block is missing the renaming of celery_result_backend to just > result_backed. > > When setting this through an environment variable, the deprecated config name > is not being used and instead the default value in the file is being used. > This is obviously remedied by the reading the UPDATING and setting the new > name, but this change has broken back compat as far as I can tell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2716: --- Affects Version/s: (was: 2.0.0) 1.10.0 Fix Version/s: 1.10.1 > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0, 1.10.1 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3117. Resolution: Fixed Fix Version/s: 2.0.0 > Add installation instructions for including unidecode GPL dependency > > > Key: AIRFLOW-3117 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Brylie Christopher Oxley >Assignee: Brylie Christopher Oxley >Priority: Trivial > Labels: newbie > Fix For: 2.0.0 > > > The installation instructions do not currently describe how to explictly > _allow_ the GPL dependency. In fairness, the documentation should describe > both scenarios, allowing and denying the GPL dependency. That way, end-users > can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency
ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/installation.rst b/docs/installation.rst index 921f2c8d46..3db48e45dd 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -19,6 +19,9 @@ You can also install Airflow with support for extra features like ``s3`` or ``po .. note:: GPL dependency One of the dependencies of Apache Airflow by default pulls in a GPL library ('unidecode'). + +If you are not concerned about the GPL dependency, export the following environment variable prior to installing airflow: ``export AIRFLOW_GPL_UNIDECODE=yes``. + In case this is a concern you can force a non GPL library by issuing ``export SLUGIFY_USES_TEXT_UNIDECODE=yes`` and then proceed with the normal installation. Please note that this needs to be specified at every upgrade. Also note that if `unidecode` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628471#comment-16628471 ] ASF GitHub Bot commented on AIRFLOW-3117: - ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/installation.rst b/docs/installation.rst index 921f2c8d46..3db48e45dd 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -19,6 +19,9 @@ You can also install Airflow with support for extra features like ``s3`` or ``po .. note:: GPL dependency One of the dependencies of Apache Airflow by default pulls in a GPL library ('unidecode'). + +If you are not concerned about the GPL dependency, export the following environment variable prior to installing airflow: ``export AIRFLOW_GPL_UNIDECODE=yes``. + In case this is a concern you can force a non GPL library by issuing ``export SLUGIFY_USES_TEXT_UNIDECODE=yes`` and then proceed with the normal installation. Please note that this needs to be specified at every upgrade. Also note that if `unidecode` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add installation instructions for including unidecode GPL dependency > > > Key: AIRFLOW-3117 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Brylie Christopher Oxley >Assignee: Brylie Christopher Oxley >Priority: Trivial > Labels: newbie > > The installation instructions do not currently describe how to explictly > _allow_ the GPL dependency. In fairness, the documentation should describe > both scenarios, allowing and denying the GPL dependency. That way, end-users > can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash
ashb commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash URL: https://github.com/apache/incubator-airflow/pull/3650#issuecomment-424648004 > If there is any continuous issue with the connection to the DB then we were expecting the external process supervisor to take care of it and restart the scheduler. Does that throw up as a different exception class? I guess my question is does this change now mask that issue and prevent the scheduler from failing over in the first case? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
ron819 commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424642655 @ashb not sure. This PR offers solution only for PostgreSQL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc
ashb commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#issuecomment-424641567 So extra thing about this - every directory is checked for an `.airflowignore`. I.e. ``` $ find dags -type f dags/.airflowignore dags/x_ignored.py dags/y_not_ignored.py dags/ignored_dir/this/whole/path/is/not/even/walked dags/lib/.airflowignore dags/lib/x_ignored.py dags/lib/y_ignored.py $ cat dags/..airflowignore ignored_dir$ x_.*\.py $ cat dags/libs/.airflowignore y_.*\.py ``` Though I've done this from memory (and sadly didn't write tests when I fixed the behaviour. Bad Ash) so a quick check I've said the right thing would be good. The key points being that the patterns are also checked against directories (and if a dir is ignored then we don't do an `os.walk` in to it, which can speed up things) and that patterns from a parent folder are also checked against the contents of sub folders.. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628426#comment-16628426 ] ASF GitHub Bot commented on AIRFLOW-3117: - brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 The installation instructions failed to mention how to proceed with the GPL dependency. For those who are not concerned by GPL, it is useful to know how to proceed with GPL dependency. ### Description Add instructions for allowing the GPL software installation during installation process. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add installation instructions for including unidecode GPL dependency > > > Key: AIRFLOW-3117 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Brylie Christopher Oxley >Assignee: Brylie Christopher Oxley >Priority: Trivial > Labels: newbie > > The installation instructions do not currently describe how to explictly > _allow_ the GPL dependency. In fairness, the documentation should describe > both scenarios, allowing and denying the GPL dependency. That way, end-users > can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency
brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 The installation instructions failed to mention how to proceed with the GPL dependency. For those who are not concerned by GPL, it is useful to know how to proceed with GPL dependency. ### Description Add instructions for allowing the GPL software installation during installation process. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
Brylie Christopher Oxley created AIRFLOW-3117: - Summary: Add installation instructions for including unidecode GPL dependency Key: AIRFLOW-3117 URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Brylie Christopher Oxley Assignee: Brylie Christopher Oxley The installation instructions do not currently describe how to explictly _allow_ the GPL dependency. In fairness, the documentation should describe both scenarios, allowing and denying the GPL dependency. That way, end-users can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] verdan commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI
verdan commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI URL: https://github.com/apache/incubator-airflow/pull/3804#issuecomment-424625200 @ashb exactly! This block was misplaced during the `npm` PR. This PR would probably be included in 2.0.0 (along with the `manage deps via npm`) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 opened a new pull request #3948: clear Val of Variable from the UI
ron819 opened a new pull request #3948: clear Val of Variable from the UI URL: https://github.com/apache/incubator-airflow/pull/3948 ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-3071) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3071 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description Allows the user to save Variable (Key,Val) where Val is empty string. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3116) Import / export connections from CLI (like for Variables)
jack created AIRFLOW-3116: - Summary: Import / export connections from CLI (like for Variables) Key: AIRFLOW-3116 URL: https://issues.apache.org/jira/browse/AIRFLOW-3116 Project: Apache Airflow Issue Type: Improvement Affects Versions: 1.10.0 Reporter: jack Variables have import/export from Json: airflow variables |-i, --import|Import variables from JSON file| |-e, --export|Export variables to JSON file| This behavior doesn't exist for connections [https://airflow.apache.org/cli.html] Add this behavior also for connections. Also requested on stackoverflow https://stackoverflow.com/questions/48636783/how-to-migrate-airflow-variables-between-dev-and-prod-environments -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3115) Add comments column to Variable page in the UI
jack created AIRFLOW-3115: - Summary: Add comments column to Variable page in the UI Key: AIRFLOW-3115 URL: https://issues.apache.org/jira/browse/AIRFLOW-3115 Project: Apache Airflow Issue Type: Task Affects Versions: 1.10.0 Reporter: jack In Admin -> Variables there are Key,Val,Is Encrypted columns Please add another column that can store comment about the variables. This will also require changes with the CLI command and the Create Variable page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ron819 commented on issue #2369: [AIRFLOW-1159] upgrading to docker >2.0
ron819 commented on issue #2369: [AIRFLOW-1159] upgrading to docker >2.0 URL: https://github.com/apache/incubator-airflow/pull/2369#issuecomment-424602966 @elipapa any progress with this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services