[GitHub] [airflow] yuqian90 commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared

2020-02-14 Thread GitBox
yuqian90 commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared URL: https://github.com/apache/airflow/pull/7276#discussion_r379783888 ## File path: airflow/ti_deps/deps/branch_dep.py ## @@ -0,0 +1,63 @@ +# +# Licensed to

[GitHub] [airflow] brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559290 How did this get tagged as a GCP-related PR?

[GitHub] [airflow] brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard removed a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488 Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (a

[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488 Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (seemingly

[GitHub] [airflow] brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard edited a comment on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559488 Also, why is [`isort` failing](https://travis-ci.org/apache/airflow/jobs/650737702#L1742) on an untouched (an

[GitHub] [airflow] brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard commented on issue #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423#issuecomment-586559290 How did this get tagged as a GCP-related PR? This is

[GitHub] [airflow] jithin97 commented on a change in pull request #7322: [AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators

2020-02-14 Thread GitBox
jithin97 commented on a change in pull request #7322: [AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators URL: https://github.com/apache/airflow/pull/7322#discussion_r379713189 ## File path: airflow/providers/google/cloud/hooks/stackdriver.py ## @@ -0,0 +1,653

[GitHub] [airflow] jithin97 commented on a change in pull request #7322: [AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators

2020-02-14 Thread GitBox
jithin97 commented on a change in pull request #7322: [AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators URL: https://github.com/apache/airflow/pull/7322#discussion_r379713197 ## File path: airflow/providers/google/cloud/operators/stackdriver.py ## @@ -0,0 +1,

[jira] [Created] (AIRFLOW-6814) Airflow Scheduler fails for any dag with MS SQL server as the backend

2020-02-14 Thread Amith Gopal (Jira)
Amith Gopal created AIRFLOW-6814: Summary: Airflow Scheduler fails for any dag with MS SQL server as the backend Key: AIRFLOW-6814 URL: https://issues.apache.org/jira/browse/AIRFLOW-6814 Project: Apac

[GitHub] [airflow] codecov-io commented on issue #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated

2020-02-14 Thread GitBox
codecov-io commented on issue #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated URL: https://github.com/apache/airflow/pull/7424#issuecomment-586545461 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7424?src=pr&el=h1) Report > Merging [#7424](https://codeco

[GitHub] [airflow] chrismclennon opened a new pull request #7425: [AIRFLOW-6801] Make use of ImportError.timezone

2020-02-14 Thread GitBox
chrismclennon opened a new pull request #7425: [AIRFLOW-6801] Make use of ImportError.timezone URL: https://github.com/apache/airflow/pull/7425 This change makes use of the `ImportError.timestamp` field. When an ImportError is added, the timestamp field is populated with the time that the

[jira] [Commented] (AIRFLOW-6801) Make use of timestamp field in ImportError object

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037384#comment-17037384 ] ASF GitHub Bot commented on AIRFLOW-6801: - chrismclennon commented on pull requ

[jira] [Work started] (AIRFLOW-6801) Make use of timestamp field in ImportError object

2020-02-14 Thread Chris McLennon (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6801 started by Chris McLennon. --- > Make use of timestamp field in ImportError object > --

[jira] [Assigned] (AIRFLOW-6801) Make use of timestamp field in ImportError object

2020-02-14 Thread Chris McLennon (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McLennon reassigned AIRFLOW-6801: --- Assignee: Chris McLennon > Make use of timestamp field in ImportError object > ---

[jira] [Commented] (AIRFLOW-6796) Serialized DAGs can be incorrectly deleted

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037382#comment-17037382 ] ASF GitHub Bot commented on AIRFLOW-6796: - MatthewRBruce commented on pull requ

[GitHub] [airflow] MatthewRBruce opened a new pull request #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated

2020-02-14 Thread GitBox
MatthewRBruce opened a new pull request #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated URL: https://github.com/apache/airflow/pull/7424 DAG serializations were previous deleted based on whether the DagFileProcessorManager had processed a particular python file.

[jira] [Commented] (AIRFLOW-3126) kubernetes executor not allowing new PV

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037375#comment-17037375 ] ASF GitHub Bot commented on AIRFLOW-3126: - brandonwillard commented on pull req

[GitHub] [airflow] brandonwillard opened a new pull request #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes

2020-02-14 Thread GitBox
brandonwillard opened a new pull request #7423: [AIRFLOW-3126] Add option to specify additional K8s volumes URL: https://github.com/apache/airflow/pull/7423 This PR introduces a new config option, `kubernetes.extra_volume_mounts`, that allows users to specify multiple Kubernetes volumes to

[jira] [Assigned] (AIRFLOW-3126) kubernetes executor not allowing new PV

2020-02-14 Thread Brandon Willard (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Willard reassigned AIRFLOW-3126: Assignee: Brandon Willard > kubernetes executor not allowing new PV > --

[jira] [Created] (AIRFLOW-6813) Error while running airflow resetdb with SQL Server as backend

2020-02-14 Thread Amith Gopal (Jira)
Amith Gopal created AIRFLOW-6813: Summary: Error while running airflow resetdb with SQL Server as backend Key: AIRFLOW-6813 URL: https://issues.apache.org/jira/browse/AIRFLOW-6813 Project: Apache Airf

[jira] [Commented] (AIRFLOW-3211) Airflow losing track of running GCP Dataproc jobs upon Airflow scheduler restart

2020-02-14 Thread Andrey Klochkov (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037342#comment-17037342 ] Andrey Klochkov commented on AIRFLOW-3211: -- The functionality added by this st

[GitHub] [airflow] digger commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library

2020-02-14 Thread GitBox
digger commented on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-586502594 @dossett, the functionality added in AIRFLOW-3211 actually broke the behavior of the dataproc hook and made a few 1.10.x

[GitHub] [airflow] digger edited a comment on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library

2020-02-14 Thread GitBox
digger edited a comment on issue #6371: [AIRFLOW-5691] Rewrite Dataproc operators to use python library URL: https://github.com/apache/airflow/pull/6371#issuecomment-586502594 @dossett, the functionality added in AIRFLOW-3211 actually broke the behavior of the dataproc hook and made a few

[GitHub] [airflow] ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared

2020-02-14 Thread GitBox
ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared URL: https://github.com/apache/airflow/pull/7276#discussion_r379649516 ## File path: airflow/models/baseoperator.py ## @@ -650,6 +651,7 @@ def deps(self) -> Set[

[GitHub] [airflow] codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

2020-02-14 Thread GitBox
codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr&el=h1) Report > Merging [#6075](https://co

[jira] [Created] (AIRFLOW-6812) KubernetesJobOperator for airflow

2020-02-14 Thread Zav Shotan (Jira)
Zav Shotan created AIRFLOW-6812: --- Summary: KubernetesJobOperator for airflow Key: AIRFLOW-6812 URL: https://issues.apache.org/jira/browse/AIRFLOW-6812 Project: Apache Airflow Issue Type: Improv

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586472520 That 8x was from comparing `time pre-commit run mypy -a` (~4.7s) to `mypy airflow docs test` (0.7s) 6.7x difference. This is on linux, and entir

[GitHub] [airflow] potiuk edited a comment on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
potiuk edited a comment on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586470454 > Could be. Would be interesting to see if mypy also chases down "delayed" imports (i.e. ones inside functions) at which point there's only so m

[GitHub] [airflow] potiuk commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
potiuk commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586470454 > Could be. Would be interesting to see if mypy also chases down "delayed" imports (i.e. ones inside functions) at which point there's only so much we

[GitHub] [airflow] potiuk commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
potiuk commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586468956 How do you compare this? where this 8x comes from? I just see one value. Is it Linux or mac? I believe on mac there is potentially huge d

[GitHub] [airflow] codecov-io edited a comment on issue #7400: [AIRFLOW-6759] Added MLEngine operator/hook to cancel MLEngine jobs

2020-02-14 Thread GitBox
codecov-io edited a comment on issue #7400: [AIRFLOW-6759] Added MLEngine operator/hook to cancel MLEngine jobs URL: https://github.com/apache/airflow/pull/7400#issuecomment-584739074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7400?src=pr&el=h1) Report > Merging [#7400](http

[GitHub] [airflow] codecov-io edited a comment on issue #7400: [AIRFLOW-6759] Added MLEngine operator/hook to cancel MLEngine jobs

2020-02-14 Thread GitBox
codecov-io edited a comment on issue #7400: [AIRFLOW-6759] Added MLEngine operator/hook to cancel MLEngine jobs URL: https://github.com/apache/airflow/pull/7400#issuecomment-584739074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7400?src=pr&el=h1) Report > Merging [#7400](http

[GitHub] [airflow] blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators

2020-02-14 Thread GitBox
blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#issuecomment-586460736 @feluelle Done! This is an automated message from the Apache Git Service. To re

[GitHub] [airflow] codecov-io commented on issue #7422: [AIRFLOW-6809] Test for presto operators

2020-02-14 Thread GitBox
codecov-io commented on issue #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#issuecomment-586458927 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7422?src=pr&el=h1) Report > Merging [#7422](https://codecov.io/gh/apache/airflow/pu

[GitHub] [airflow] ashb edited a comment on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb edited a comment on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586457351 @potiuk Good idea on require_serial -- that has the same effect for all files, but makes a single file "work" and it is still marginally quicker f

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586457823 Just for the record: It seems that "dockerising" it adds about an 8x overhead though :( ``` $ time mypy airflow docs/ tests Success: no issu

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586457351 @potiuk Good idea on require_serial -- that has the same effect for all files, but makes a single file "work" and it is still marginally quicker for a s

[jira] [Updated] (AIRFLOW-6802) scheduler not honoring dag.max_active_run config

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-6802: --- Fix Version/s: (was: 2.0.0) 1.10.10 > scheduler not honoring

[GitHub] [airflow] codecov-io edited a comment on issue #7353: [AIRFLOW-6685] ThresholdCheckOperator

2020-02-14 Thread GitBox
codecov-io edited a comment on issue #7353: [AIRFLOW-6685] ThresholdCheckOperator URL: https://github.com/apache/airflow/pull/7353#issuecomment-582190961 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr&el=h1) Report > Merging [#7353](https://codecov.io/gh/apache/airfl

[jira] [Updated] (AIRFLOW-6779) Schedule stopped scheduling tasks with concurrency limit or depends on past

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-6779: --- Parent: AIRFLOW-3607 Issue Type: Sub-task (was: Bug) > Schedule stopped sc

[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037261#comment-17037261 ] ASF GitHub Bot commented on AIRFLOW-3607: - ashb commented on pull request #7402

[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037262#comment-17037262 ] ASF subversion and git services commented on AIRFLOW-3607: -- Co

[GitHub] [airflow] ashb merged pull request #7402: [AIRFLOW-3607] fix scheduler bug related to task concurrency and depends on past

2020-02-14 Thread GitBox
ashb merged pull request #7402: [AIRFLOW-3607] fix scheduler bug related to task concurrency and depends on past URL: https://github.com/apache/airflow/pull/7402 This is an automated message from the Apache Git Service. To r

[GitHub] [airflow] ashb commented on issue #7402: [AIRFLOW-6779] fix scheduler bug related to task concurrency and depends on past

2020-02-14 Thread GitBox
ashb commented on issue #7402: [AIRFLOW-6779] fix scheduler bug related to task concurrency and depends on past URL: https://github.com/apache/airflow/pull/7402#issuecomment-586451932 I've re-targeted this PR against AIRFLOW-3607 as it's a bug fix to a commit that has not yet been included

[GitHub] [airflow] peter-volkov edited a comment on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov edited a comment on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#issuecomment-586446523 @nuclearpinguin thank you. What about estimated release time? As I understand. by default it will be published in Ai

[GitHub] [airflow] peter-volkov commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#issuecomment-586446523 @nuclearpinguin thank you. What about estimated release time? As I understand. by default it will be published in Airflow 2

[jira] [Work stopped] (AIRFLOW-6799) webgui cannot display all tasks

2020-02-14 Thread Soeren Laursen (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6799 stopped by Soeren Laursen. --- > webgui cannot display all tasks > --- > >

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037216#comment-17037216 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037215#comment-17037215 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037213#comment-17037213 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037217#comment-17037217 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037219#comment-17037219 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[GitHub] [airflow] nuclearpinguin commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#issuecomment-586426722 @peter-volkov thanks for the contribution! 🚀 This is an aut

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037214#comment-17037214 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037212#comment-17037212 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037218#comment-17037218 ] ASF subversion and git services commented on AIRFLOW-6531: -- Co

[jira] [Resolved] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread Tomasz Urbaszek (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6531. -- Fix Version/s: 2.0.0 Resolution: Done > Add Yandex.Cloud support in Airflow >

[jira] [Commented] (AIRFLOW-6531) Add Yandex.Cloud support in Airflow

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037211#comment-17037211 ] ASF GitHub Bot commented on AIRFLOW-6531: - nuclearpinguin commented on pull req

[GitHub] [airflow] nuclearpinguin merged pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin merged pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252 This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
boring-cyborg[bot] commented on issue #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#issuecomment-586425903 Awesome work, congrats on your first merged pull request! --

[GitHub] [airflow] vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow

2020-02-14 Thread GitBox
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add singularity to airflow URL: https://github.com/apache/airflow/pull/7191#issuecomment-586424350 Questions: - where does example_singularity_operator.py go - it used to be under contrib/example_dags but now, that en

[jira] [Created] (AIRFLOW-6811) Workflows randomply fail because of ERROR - Unknown error in KubernetesJobWatcher

2020-02-14 Thread Natalia Ogden (Jira)
Natalia Ogden created AIRFLOW-6811: -- Summary: Workflows randomply fail because of ERROR - Unknown error in KubernetesJobWatcher Key: AIRFLOW-6811 URL: https://issues.apache.org/jira/browse/AIRFLOW-6811

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586405375 Could be. Would be interesting to see if mypy also chases down "delayed" imports (i.e. ones inside functions) at which point there's only so much we can

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379555857 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -0,

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379555770 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -0,

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379555675 ## File path: airflow/contrib/hooks/yandexcloud_dataproc_hook.py ## @@ -0,0 +1,571 @@ +# -*

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379555724 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -0,

[GitHub] [airflow] potiuk edited a comment on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
potiuk edited a comment on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586372061 And BTW this is totally expected with our codebase if you think of it. It is yet another consequence of our airflow.__init__.py which immediate

[jira] [Resolved] (AIRFLOW-6795) serialized_dag table's data column text type is too small for mysql

2020-02-14 Thread Nathaniel Ritholtz (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathaniel Ritholtz resolved AIRFLOW-6795. - Resolution: Won't Fix I just bumped our MySQL instance to v5.7+ to use JSON dat

[GitHub] [airflow] potiuk commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
potiuk commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586372061 And BTW this is totally expect with our codebase if you think of it. It is yet another consequence of our airflow.__init__.py which immediately loads

[jira] [Commented] (AIRFLOW-6795) serialized_dag table's data column text type is too small for mysql

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037109#comment-17037109 ] ASF GitHub Bot commented on AIRFLOW-6795: - nritholtz commented on pull request

[GitHub] [airflow] nritholtz closed pull request #7414: [AIRFLOW-6795] Increase text size on data column in serialized_dag for MySQL

2020-02-14 Thread GitBox
nritholtz closed pull request #7414: [AIRFLOW-6795] Increase text size on data column in serialized_dag for MySQL URL: https://github.com/apache/airflow/pull/7414 This is an automated message from the Apache Git Service. To

[jira] [Commented] (AIRFLOW-4449) Default permissions for custom roles

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037106#comment-17037106 ] ASF GitHub Bot commented on AIRFLOW-4449: - stale[bot] commented on pull request

[GitHub] [airflow] stale[bot] closed pull request #5812: [AIRFLOW-4449] updated default permission for custom roles to 'Viewer'

2020-02-14 Thread GitBox
stale[bot] closed pull request #5812: [AIRFLOW-4449] updated default permission for custom roles to 'Viewer' URL: https://github.com/apache/airflow/pull/5812 This is an automated message from the Apache Git Service. To respo

[jira] [Created] (AIRFLOW-6810) KubernetesPodOperator pod is completed but xcom side car is stuck

2020-02-14 Thread Maxence Cramet (Jira)
Maxence Cramet created AIRFLOW-6810: --- Summary: KubernetesPodOperator pod is completed but xcom side car is stuck Key: AIRFLOW-6810 URL: https://issues.apache.org/jira/browse/AIRFLOW-6810 Project: Ap

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586350360 Sure, I'll test that out. This is an automated message from the Apache Git Service. To r

[GitHub] [airflow] ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared

2020-02-14 Thread GitBox
ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared URL: https://github.com/apache/airflow/pull/7276#discussion_r379470709 ## File path: tests/operators/test_python.py ## @@ -396,7 +397,7 @@ def test_without_dag_r

[GitHub] [airflow] ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared

2020-02-14 Thread GitBox
ashb commented on a change in pull request #7276: [AIRFLOW-5391] Do not run skipped tasks when they are cleared URL: https://github.com/apache/airflow/pull/7276#discussion_r379470137 ## File path: airflow/ti_deps/deps/branch_dep.py ## @@ -0,0 +1,63 @@ +# +# Licensed to the

[GitHub] [airflow] blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators

2020-02-14 Thread GitBox
blcksrx commented on issue #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#issuecomment-586314391 > Let me know when you have fixed the static checks / linting issues. :) thanks. can you check my last PR until I fix this =

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379461908 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -0,

[GitHub] [airflow] peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
peter-volkov commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379461908 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -0,

[GitHub] [airflow] feluelle commented on issue #7422: [AIRFLOW-6809] Test for presto operators

2020-02-14 Thread GitBox
feluelle commented on issue #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#issuecomment-586311321 Let me know when you have fixed the static checks / linting issues. :) This is

[GitHub] [airflow] feluelle commented on a change in pull request #7422: [AIRFLOW-6809] Test for presto operators

2020-02-14 Thread GitBox
feluelle commented on a change in pull request #7422: [AIRFLOW-6809] Test for presto operators URL: https://github.com/apache/airflow/pull/7422#discussion_r379456567 ## File path: tests/providers/presto/operators/test_presto_check.py ## @@ -0,0 +1,224 @@ +# +# Licensed to

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379456748 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379455746 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379455746 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support

2020-02-14 Thread GitBox
nuclearpinguin commented on a change in pull request #7252: [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support URL: https://github.com/apache/airflow/pull/7252#discussion_r379454451 ## File path: airflow/providers/yandex/operators/yandexcloud_dataproc_operator.py ## @@ -

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7363: [AIRFLOW-6730] Use total_seconds instead of seconds

2020-02-14 Thread GitBox
nuclearpinguin commented on a change in pull request #7363: [AIRFLOW-6730] Use total_seconds instead of seconds URL: https://github.com/apache/airflow/pull/7363#discussion_r379452203 ## File path: tests/providers/google/cloud/operators/test_dataproc.py ## @@ -104,7 +104,7

[jira] [Comment Edited] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2020-02-14 Thread wei chea (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036815#comment-17036815 ] wei chea edited comment on AIRFLOW-2219 at 2/14/20 2:03 PM:

[jira] [Comment Edited] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036991#comment-17036991 ] Ash Berlin-Taylor edited comment on AIRFLOW-2219 at 2/14/20 2:02 PM:

[jira] [Commented] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036991#comment-17036991 ] Ash Berlin-Taylor commented on AIRFLOW-2219: [~weichea] Yours is a differen

[jira] [Issue Comment Deleted] (AIRFLOW-2219) Race condition to DagRun.verify_integrity between Scheduler and Webserver

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2219: --- Comment: was deleted (was: i get same error using trigger_dag in cli:   env local

[jira] [Closed] (AIRFLOW-1370) Scheduler is crashing because of IntegrityError

2020-02-14 Thread Ash Berlin-Taylor (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor closed AIRFLOW-1370. -- Resolution: Cannot Reproduce Not happening in the scheduler anymore. Issues with thi

[GitHub] [airflow] feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

2020-02-14 Thread GitBox
feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r379362800 ## File path: airflow/providers/amazon/aws/hooks/athena.py ## @@ -132,7 +136,37 @@ def

[GitHub] [airflow] feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

2020-02-14 Thread GitBox
feluelle commented on a change in pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#discussion_r379362690 ## File path: airflow/providers/amazon/aws/hooks/athena.py ## @@ -116,7 +120,7 @@ def

[GitHub] [airflow] ashb commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image.

2020-02-14 Thread GitBox
ashb commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. URL: https://github.com/apache/airflow/pull/6538#discussion_r379438095 ## File path: Dockerfile ## @@ -357,6 +341,29 @@ RUN chmod +x "${HOME}/run-tests-complete"

[GitHub] [airflow] ashb commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image.

2020-02-14 Thread GitBox
ashb commented on a change in pull request #6538: [AIRFLOW-5886] Selective copying of sources in Docker image. URL: https://github.com/apache/airflow/pull/6538#discussion_r379436725 ## File path: Dockerfile ## @@ -357,6 +341,29 @@ RUN chmod +x "${HOME}/run-tests-complete"

[GitHub] [airflow] ashb commented on issue #7421: [AIRFLOW-XXXX] Speed up mypy runs.

2020-02-14 Thread GitBox
ashb commented on issue #7421: [AIRFLOW-] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421#issuecomment-586290473 Seems to give a modest speed up on Travis (25s to 19s) This is an automated message from

[jira] [Commented] (AIRFLOW-6809) Test for presto operators

2020-02-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036979#comment-17036979 ] ASF GitHub Bot commented on AIRFLOW-6809: - blcksrx commented on pull request #7

[GitHub] [airflow] blcksrx opened a new pull request #7422: [AIRFLOW-6809] - Test for presto operators

2020-02-14 Thread GitBox
blcksrx opened a new pull request #7422: [AIRFLOW-6809] - Test for presto operators URL: https://github.com/apache/airflow/pull/7422 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR:

  1   2   >