[GitHub] [airflow] driquelme opened a new issue #9186: Add SRV DNS record support to http connections

2020-06-09 Thread GitBox
driquelme opened a new issue #9186: URL: https://github.com/apache/airflow/issues/9186 **Description** SRV records are DNS records that resolve to a hostname and a port. Being able to specify http connections with SRV resolution support is useful in contexts where external

[GitHub] [airflow] mik-laj opened a new pull request #9194: Remove httplib2 from Google requirements

2020-06-09 Thread GitBox
mik-laj opened a new pull request #9194: URL: https://github.com/apache/airflow/pull/9194 We don't use this library directly, so we can rely on the requirements of other libraries. On the other hand, this requirement conflicts with Apache Beam v2.19 ('httplib2>=0.8,<=0.12.0',). It is

[GitHub] [airflow] languitar commented on issue #8629: DockerOperator cannot pull image when docker client uses json-file log driver

2020-06-09 Thread GitBox
languitar commented on issue #8629: URL: https://github.com/apache/airflow/issues/8629#issuecomment-641282826 Does anyone know when this will end up in a stable release? This is an automated message from the Apache Git

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129581#comment-17129581 ] ASF GitHub Bot commented on AIRFLOW-3973: - Fokko commented on pull request #4797: URL:

[GitHub] [airflow] Fokko commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
Fokko commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640588665 @eeshugerman please make a PR to the `v1-10-test` branch and I'll merge it. This is an automated message from the

[GitHub] [airflow] ashb merged pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
ashb merged pull request #9180: URL: https://github.com/apache/airflow/pull/9180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] jimmajure opened a new issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

2020-06-09 Thread GitBox
jimmajure opened a new issue #9181: URL: https://github.com/apache/airflow/issues/9181 **Description** In the KubernetesExecutor configuration, allow the DAGs, logs, plugins folders to be supported by a single PersistentVolumeClaim. Currently the configuration forces a separate

[jira] [Commented] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129576#comment-17129576 ] ASF GitHub Bot commented on AIRFLOW-6994: - stale[bot] commented on pull request #7637: URL:

[GitHub] [airflow] stale[bot] commented on pull request #7637: [AIRFLOW-6994] SparkSubmitOperator re-launches spark driver even when original driver still running

2020-06-09 Thread GitBox
stale[bot] commented on pull request #7637: URL: https://github.com/apache/airflow/pull/7637#issuecomment-64410 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitHub] [airflow] dimon222 commented on issue #9148: Retries are causing constraint violation (Airflow 1.10.10)

2020-06-09 Thread GitBox
dimon222 commented on issue #9148: URL: https://github.com/apache/airflow/issues/9148#issuecomment-641335335 > @dimon222 Can you try setting `max_num_rendered_ti_fields_per_task` under `[core]` section to 1 and see it that helps for now ? Not helping, receiving same error. I'm

[GitHub] [airflow] potiuk merged pull request #9188: Fixes failure of the build scripts when remote repo does not exist

2020-06-09 Thread GitBox
potiuk merged pull request #9188: URL: https://github.com/apache/airflow/pull/9188 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] mik-laj merged pull request #9177: Add PR/issue note in Contribution Workflow Example

2020-06-09 Thread GitBox
mik-laj merged pull request #9177: URL: https://github.com/apache/airflow/pull/9177 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] mik-laj commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r437198960 ## File path: airflow/providers/google/cloud/hooks/cloud_build.py ## @@ -47,90 +47,528 @@ class CloudBuildHook(GoogleBaseHook): :type delegate_to:

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r437473652 ## File path: airflow/serialization/serialized_objects.py ## @@ -297,7 +297,7 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization):

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129572#comment-17129572 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on a change in pull request #8575:

[GitHub] [airflow] OmairK commented on a change in pull request #9097: [WIP] Schema and CRUD endpoints Pool

2020-06-09 Thread GitBox
OmairK commented on a change in pull request #9097: URL: https://github.com/apache/airflow/pull/9097#discussion_r437471258 ## File path: airflow/api_connexion/endpoints/pool_endpoint.py ## @@ -26,18 +30,33 @@ def delete_pool(): raise NotImplementedError("Not implemented

[GitHub] [airflow] mik-laj commented on a change in pull request #9097: [WIP] Schema and CRUD endpoints Pool

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9097: URL: https://github.com/apache/airflow/pull/9097#discussion_r437017632 ## File path: airflow/api_connexion/endpoints/pool_endpoint.py ## @@ -26,18 +30,33 @@ def delete_pool(): raise NotImplementedError("Not implemented

[GitHub] [airflow] JeffryMAC commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
JeffryMAC commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436784765 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *',

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129559#comment-17129559 ] ASF GitHub Bot commented on AIRFLOW-7014: - liuyonghengheng commented on a change in pull

[GitHub] [airflow] liuyonghengheng commented on a change in pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator

2020-06-09 Thread GitBox
liuyonghengheng commented on a change in pull request #9149: URL: https://github.com/apache/airflow/pull/9149#discussion_r437343515 ## File path: airflow/providers/apache/kylin/operators/kylin.py ## @@ -0,0 +1,166 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] turbaszek merged pull request #9087: Query TaskReschedule only if task is UP_FOR_RESCHEDULE

2020-06-09 Thread GitBox
turbaszek merged pull request #9087: URL: https://github.com/apache/airflow/pull/9087 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] the-horhe opened a new issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
the-horhe opened a new issue #9190: URL: https://github.com/apache/airflow/issues/9190 **Apache Airflow version**: 1.10.10 **Environment**: official docker image apache/airflow:1.10.10, docker api version: 1.40 **What happened**: Don't see any container output in task logs

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9153: [WIP] add readonly endpoints for dagruns

2020-06-09 Thread GitBox
ephraimbuddy commented on a change in pull request #9153: URL: https://github.com/apache/airflow/pull/9153#discussion_r436643921 ## File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py ## @@ -29,26 +35,471 @@ def setUpClass(cls) -> None: def setUp(self) ->

[GitHub] [airflow] ashb opened a new pull request #9191: Update AWS connection example to show how to set from env var

2020-06-09 Thread GitBox
ashb opened a new pull request #9191: URL: https://github.com/apache/airflow/pull/9191 The trailing `@` wasn't obvious/documented anywhere (and took me some trial and error to work out) so to save time for the next person let's add it to the docs --- Make sure to mark

[GitHub] [airflow] turbaszek commented on a change in pull request #9054: Add MySqlToS3Operator

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #9054: URL: https://github.com/apache/airflow/pull/9054#discussion_r437208286 ## File path: airflow/providers/amazon/aws/operators/mysql_to_s3.py ## @@ -0,0 +1,126 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] mik-laj merged pull request #9169: Allows using private endpoints in GKEStartPodOperator

2020-06-09 Thread GitBox
mik-laj merged pull request #9169: URL: https://github.com/apache/airflow/pull/9169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] ashb commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ashb commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-640741641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] potiuk commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
potiuk commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640687352 Just a small update: I think merge PR should go to v1-10-stable - in test we are usually cherry-pick stuff (I have 23 cherry-picked commits to push soon) so better to keep it

[GitHub] [airflow] feluelle commented on a change in pull request #9195: Update pre-commit-hooks repo version

2020-06-09 Thread GitBox
feluelle commented on a change in pull request #9195: URL: https://github.com/apache/airflow/pull/9195#discussion_r437508400 ## File path: .pre-commit-config.yaml ## @@ -159,15 +159,23 @@ metastore_browser/templates/.*\\.html$|.*\\.jinja2" hooks: - id:

[GitHub] [airflow] ashb opened a new pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
ashb opened a new pull request #9180: URL: https://github.com/apache/airflow/pull/9180 Most things already use the `Markup` class to correctly escape problem areas, this commit just fixes the last instances so that we can assert that `|safe` is never used. --- Make sure

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129539#comment-17129539 ] ASF GitHub Bot commented on AIRFLOW-3973: - potiuk commented on pull request #4797: URL:

[GitHub] [airflow] potiuk opened a new pull request #9188: Fixes failure of the build scripts when remote repo does not exist

2020-06-09 Thread GitBox
potiuk opened a new pull request #9188: URL: https://github.com/apache/airflow/pull/9188 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] ashb commented on pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
ashb commented on pull request #9187: URL: https://github.com/apache/airflow/pull/9187#issuecomment-641273689 > I think we should also set conn_id as non-nullable Yes, agreed. Let's do that in one migration. This is

[GitHub] [airflow] mik-laj commented on a change in pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9180: URL: https://github.com/apache/airflow/pull/9180#discussion_r436887040 ## File path: airflow/www/views.py ## @@ -631,11 +630,9 @@ def rendered(self): for template_field in task.template_fields: content

[GitHub] [airflow] edejong commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-06-09 Thread GitBox
edejong commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-641387021 I should really check GitHub more often, I only saw the notification now. Let me know if I understand the question correctly: should the BigQueryHook's interface rely on

[GitHub] [airflow] mjpieters opened a new issue #9185: Include plugin information in the version page

2020-06-09 Thread GitBox
mjpieters opened a new issue #9185: URL: https://github.com/apache/airflow/issues/9185 **Description** Please include plug-in information (see below) in the Airflow "About -> Version" information page in the web interface. **Use case / motivation** Plug-ins have a lot

[GitHub] [airflow] randr97 commented on a change in pull request #9170: [WIP] Read only endpoint for XCom #8134

2020-06-09 Thread GitBox
randr97 commented on a change in pull request #9170: URL: https://github.com/apache/airflow/pull/9170#discussion_r437123042 ## File path: airflow/api_connexion/schemas/xcom_schema.py ## @@ -0,0 +1,67 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] crhyatt opened a new pull request #9184: Upgrade pendulum to latest major version ~2.0

2020-06-09 Thread GitBox
crhyatt opened a new pull request #9184: URL: https://github.com/apache/airflow/pull/9184 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] potiuk commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
potiuk commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-640799787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r437204483 ## File path: airflow/operators/python.py ## @@ -145,6 +149,131 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] chamcca commented on pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-06-09 Thread GitBox
chamcca commented on pull request #8749: URL: https://github.com/apache/airflow/pull/8749#issuecomment-640603607 @feluelle any additional comments/recommendations for this PR? This is an automated message from the Apache

[GitHub] [airflow] mik-laj commented on pull request #9067: [AIRFLOW-8608] Make conn_id unique in Connections table

2020-06-09 Thread GitBox
mik-laj commented on pull request #9067: URL: https://github.com/apache/airflow/pull/9067#issuecomment-640912099 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[airflow] branch master updated (b762763 -> 7ad827f)

2020-06-09 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from b762763 Query TaskReschedule only if task is UP_FOR_RESCHEDULE (#9087) add 7ad827f Set conn_type as

[GitHub] [airflow] turbaszek commented on pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek commented on pull request #8805: URL: https://github.com/apache/airflow/pull/8805#issuecomment-641253096 Meh. The serialization doesn't work because `__call__` has other signature than `BaseOperator.__init__`. I think this could be easily solved by doing `super().__call__()`

[GitHub] [airflow] ashb merged pull request #9178: Validate only task commands are run by executors

2020-06-09 Thread GitBox
ashb merged pull request #9178: URL: https://github.com/apache/airflow/pull/9178 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] JavierLopezT commented on a change in pull request #9054: Add MySqlToS3Operator

2020-06-09 Thread GitBox
JavierLopezT commented on a change in pull request #9054: URL: https://github.com/apache/airflow/pull/9054#discussion_r437244791 ## File path: airflow/providers/amazon/aws/operators/mysql_to_s3.py ## @@ -0,0 +1,126 @@ +# +# Licensed to the Apache Software Foundation (ASF)

[GitHub] [airflow] ChristianYeah commented on issue #9131: max_active_runs failed in production env with multiple workers

2020-06-09 Thread GitBox
ChristianYeah commented on issue #9131: URL: https://github.com/apache/airflow/issues/9131#issuecomment-641222683 I’ll duplicate the production env to the test later later , see if it will be reproduced. 发自我的iPhone -- Original --

[GitHub] [airflow] mik-laj commented on pull request #9124: [AIRFLOW-9099] Merging multiple sql operators

2020-06-09 Thread GitBox
mik-laj commented on pull request #9124: URL: https://github.com/apache/airflow/pull/9124#issuecomment-641106425 Can you add these cases to the list of changed classes? The file name suggests something different, but that's a different problem. Here we have all the name changes that we

[GitHub] [airflow] mik-laj commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437254077 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software

[GitHub] [airflow] ashb opened a new pull request #9178: Validate only task commands are run by executors

2020-06-09 Thread GitBox
ashb opened a new pull request #9178: URL: https://github.com/apache/airflow/pull/9178 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] turbaszek commented on pull request #8954: Wait for pipeline state in Data Fusion operators

2020-06-09 Thread GitBox
turbaszek commented on pull request #8954: URL: https://github.com/apache/airflow/pull/8954#issuecomment-641147473 Hi @jaketf @sreevatsanraman what should we do to move this forward? This is an automated message from the

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9185: Include plugin information in the version page

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9185: URL: https://github.com/apache/airflow/issues/9185#issuecomment-640820162 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] ashb merged pull request #9191: Update AWS connection example to show how to set from env var

2020-06-09 Thread GitBox
ashb merged pull request #9191: URL: https://github.com/apache/airflow/pull/9191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] nullhack opened a new pull request #9179: Feature - add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack opened a new pull request #9179: URL: https://github.com/apache/airflow/pull/9179 **Description** The only reason I can think to not include `minutely` on `schedule_interval`, is to keep consistency against `cron`. As we can see, all other relevant shortcuts are already

[GitHub] [airflow] mik-laj commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436674133 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *', Review

[GitHub] [airflow] mik-laj opened a new pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
mik-laj opened a new pull request #9187: URL: https://github.com/apache/airflow/pull/9187 The `conn_type` column in the `connection` table must contain content. Previously, this rule was enforced by application logic, but was not enforced by the database schema. Before: ```

[GitHub] [airflow] nullhack commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436736580 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *', Review

[GitHub] [airflow] ashb commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
ashb commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437393065 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software

[GitHub] [airflow] nullhack commented on pull request #9179: Feature - add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack commented on pull request #9179: URL: https://github.com/apache/airflow/pull/9179#issuecomment-640585047 @mik-laj sorry, don't know any other maintainers yet to ask for review XD may I ask you to review (or tag someone to)? It's a small change, but my colleagues liked the

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129502#comment-17129502 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman commented on pull request #4797: URL:

[jira] [Commented] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129500#comment-17129500 ] ASF GitHub Bot commented on AIRFLOW-6994: - tooptoop4 commented on pull request #7637: URL:

[GitHub] [airflow] eeshugerman edited a comment on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
eeshugerman edited a comment on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640745946 A cherry-pick/rebase was required to get a clean PR to `v1-10-stable` or `v1-10-test`, so wasn't sure which to do? I opened a PR for each, feel free to close one or

[GitHub] [airflow] tooptoop4 commented on pull request #7637: [AIRFLOW-6994] SparkSubmitOperator re-launches spark driver even when original driver still running

2020-06-09 Thread GitBox
tooptoop4 commented on pull request #7637: URL: https://github.com/apache/airflow/pull/7637#issuecomment-641116742 crispy This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129498#comment-17129498 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman edited a comment on pull request #4797:

[GitHub] [airflow] eeshugerman commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
eeshugerman commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640745946 A cherry-pick/rebase was required to get a clean PR to `v1-10-stable` or `v1-10-test`, so wasn't sure which to do? I opened a PR for each, feel free to close one or the

[GitHub] [airflow] nullhack edited a comment on pull request #9165: Fix DockerOperator xcom

2020-06-09 Thread GitBox
nullhack edited a comment on pull request #9165: URL: https://github.com/apache/airflow/pull/9165#issuecomment-640226947 @mik-laj Thank you for the suggestions. I wrote some example on the issue Itself (https://github.com/apache/airflow/issues/9164) if you need It to test now. I'll

[GitHub] [airflow] feluelle opened a new pull request #9195: Update pre-commit-hooks repo version

2020-06-09 Thread GitBox
feluelle opened a new pull request #9195: URL: https://github.com/apache/airflow/pull/9195 - use official isort pre-commit-hook - use official yamllint pre-commit-hook --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context

[GitHub] [airflow] turbaszek removed a comment on pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek removed a comment on pull request #8805: URL: https://github.com/apache/airflow/pull/8805#issuecomment-641253096 Meh. The serialization doesn't work because `__call__` has other signature than `BaseOperator.__init__`. I think this could be easily solved by doing

[GitHub] [airflow] ash commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ash commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-641062231 Guys. This is very annoying to get email notifications when you mistype the name of one of you. This is an

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641100635 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] potiuk merged pull request #9167: Improved cloud tool available in the trimmed down CI container

2020-06-09 Thread GitBox
potiuk merged pull request #9167: URL: https://github.com/apache/airflow/pull/9167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] ashb commented on pull request #9170: [WIP] Read only endpoint for XCom #8134

2020-06-09 Thread GitBox
ashb commented on pull request #9170: URL: https://github.com/apache/airflow/pull/9170#issuecomment-641274716 @mik-laj Are you happy with https://github.com/apache/airflow/pull/9170/commits/c973a0710a6785b043d1d20ee9e09eb96cecbc86?

[GitHub] [airflow] crhyatt closed pull request #8500: [Airflow-3391] Upgrade pendulum to latest major version

2020-06-09 Thread GitBox
crhyatt closed pull request #8500: URL: https://github.com/apache/airflow/pull/8500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] madison-ookla commented on pull request #9174: Don't use the term "whitelist" - language matters

2020-06-09 Thread GitBox
madison-ookla commented on pull request #9174: URL: https://github.com/apache/airflow/pull/9174#issuecomment-640719783 Thanks Ash :) This is great to see! This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] Acehaidrey commented on pull request #8598: make hive macros return string type vs bytes

2020-06-09 Thread GitBox
Acehaidrey commented on pull request #8598: URL: https://github.com/apache/airflow/pull/8598#issuecomment-640870324 This change is actually done using branch v1.10-stable. So it works as is here @ashb . Good question. So no concern there either

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129483#comment-17129483 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman opened a new pull request #9183: URL:

[GitHub] [airflow] turbaszek commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437240866 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software

[GitHub] [airflow] eeshugerman opened a new pull request #9183: [AIRFLOW-3973] Commit after each alembic migration

2020-06-09 Thread GitBox
eeshugerman opened a new pull request #9183: URL: https://github.com/apache/airflow/pull/9183 If `Variable`s are used in DAGs, and Postgres is used for the internal database, a fresh `$ airflow initdb` (or `$ airflow resetdb`) spams the logs with error messages (but does not fail).

[GitHub] [airflow] zikun commented on pull request #9189: Support additional apt dependencies

2020-06-09 Thread GitBox
zikun commented on pull request #9189: URL: https://github.com/apache/airflow/pull/9189#issuecomment-641077907 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] feluelle commented on a change in pull request #8877: Add S3ToRedshift example dag and system test

2020-06-09 Thread GitBox
feluelle commented on a change in pull request #8877: URL: https://github.com/apache/airflow/pull/8877#discussion_r437309397 ## File path: tests/providers/amazon/aws/operators/test_s3_to_redshift_system.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] potiuk commented on pull request #8974: Add VaultHook to hashicorp provider

2020-06-09 Thread GitBox
potiuk commented on pull request #8974: URL: https://github.com/apache/airflow/pull/8974#issuecomment-641179952 @kaxil :) ? This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] dinigo opened a new issue #9193: Celery 4.4.4 (known bug) lacks "future" module thus breaks airflow

2020-06-09 Thread GitBox
dinigo opened a new issue #9193: URL: https://github.com/apache/airflow/issues/9193 Hello. We are experiencing some issues in local development due to a celery using a module that it doesn't install anymore. This happened when upping dependency version of celery 4.4.3 to 4.4.4

[GitHub] [airflow] mik-laj commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread GitBox
mik-laj commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-641094098 I still have the following error: ``` /airflow/logs/previous_runs/2020-06-09_07_26_35/example_gcp_cloud_build/create_build_from_repo/2020-06-08T00:00:00+00:00/1.log ```

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9186: Add SRV DNS record support to http connections

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9186: URL: https://github.com/apache/airflow/issues/9186#issuecomment-640844130 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129466#comment-17129466 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on pull request #8575: URL:

[GitHub] [airflow] ashb commented on a change in pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ashb commented on a change in pull request #9162: URL: https://github.com/apache/airflow/pull/9162#discussion_r436843199 ## File path: Dockerfile ## @@ -153,6 +153,23 @@ ENV PIP_VERSION=${PIP_VERSION} RUN pip install --upgrade pip==${PIP_VERSION} +ARG

[GitHub] [airflow] svetli-n commented on issue #9131: max_active_runs failed in production env with multiple workers

2020-06-09 Thread GitBox
svetli-n commented on issue #9131: URL: https://github.com/apache/airflow/issues/9131#issuecomment-641221074 @ChristianYeah Thanks. I can't reproduce it though. Feel free to share full dag run file including deafult settings and tasks, as well as how you trigger the dag run.

[GitHub] [airflow] nullhack edited a comment on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
nullhack edited a comment on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641245736 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] turbaszek commented on a change in pull request #8877: Add S3ToRedshift example dag and system test

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8877: URL: https://github.com/apache/airflow/pull/8877#discussion_r437211600 ## File path: airflow/providers/amazon/aws/example_dags/example_s3_to_redshift.py ## @@ -41,13 +42,26 @@ schedule_interval=None,

[GitHub] [airflow] nullhack commented on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
nullhack commented on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641245736 This can be related to a recently merged [PR](https://github.com/apache/airflow/pull/9173). I tried to run on the docker image `apache/airflow:master-ci` and the output was

[GitHub] [airflow] crhyatt commented on pull request #8500: [Airflow-3391] Upgrade pendulum to latest major version

2020-06-09 Thread GitBox
crhyatt commented on pull request #8500: URL: https://github.com/apache/airflow/pull/8500#issuecomment-640751169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] nehiljain commented on issue #9015: Update to Pendulum 2.1.0

2020-06-09 Thread GitBox
nehiljain commented on issue #9015: URL: https://github.com/apache/airflow/issues/9015#issuecomment-640791485 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129452#comment-17129452 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman opened a new pull request #9182: URL:

[GitHub] [airflow] tooptoop4 commented on issue #9166: DAG run fails when task instance is still running

2020-06-09 Thread GitBox
tooptoop4 commented on issue #9166: URL: https://github.com/apache/airflow/issues/9166#issuecomment-640915884 https://github.com/apache/airflow/pull/8082 might help This is an automated message from the Apache Git Service.

[GitHub] [airflow] eeshugerman opened a new pull request #9182: [AIRFLOW-3973] Commit after each alembic migration

2020-06-09 Thread GitBox
eeshugerman opened a new pull request #9182: URL: https://github.com/apache/airflow/pull/9182 If `Variable`s are used in DAGs, and Postgres is used for the internal database, a fresh `$ airflow initdb` (or `$ airflow resetdb`) spams the logs with error messages (but does not fail).

[GitHub] [airflow] thesuperzapper commented on a change in pull request #9133: Include missing RBAC roles

2020-06-09 Thread GitBox
thesuperzapper commented on a change in pull request #9133: URL: https://github.com/apache/airflow/pull/9133#discussion_r437078140 ## File path: tests/www/test_views.py ## @@ -1983,7 +1983,7 @@ def test_failed_success(self): past="false", ) resp

[GitHub] [airflow] mik-laj commented on a change in pull request #9153: [WIP] add readonly endpoints for dagruns

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9153: URL: https://github.com/apache/airflow/pull/9153#discussion_r436645575 ## File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py ## @@ -29,26 +35,471 @@ def setUpClass(cls) -> None: def setUp(self) -> None:

[GitHub] [airflow] zikun opened a new pull request #9189: Support additional apt dependencies

2020-06-09 Thread GitBox
zikun opened a new pull request #9189: URL: https://github.com/apache/airflow/pull/9189 Support ADDITONAL_DEV_DEPS and ADDITONAL_RUNTIME_DEPS build arguments in both CI and production Dockerfile. Related to: #8872 --- Make sure to mark the boxes below before creating PR:

[GitHub] [airflow] ashb commented on a change in pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
ashb commented on a change in pull request #9180: URL: https://github.com/apache/airflow/pull/9180#discussion_r436734448 ## File path: airflow/www/templates/airflow/dag.html ## @@ -427,8 +427,8 @@ }); subdag_id = sd; execution_date = d; -

[GitHub] [airflow] mik-laj commented on a change in pull request #9095: [WIP]Add readonly connection endpoints

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9095: URL: https://github.com/apache/airflow/pull/9095#discussion_r437016593 ## File path: airflow/api_connexion/schemas/base_schema.py ## @@ -0,0 +1,79 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

<    1   2   3   >