[GitHub] [airflow] turbaszek commented on pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
turbaszek commented on pull request #9038: URL: https://github.com/apache/airflow/pull/9038#issuecomment-635115335 Still the same, old error `Couldn't resolve host 'github.com'` :< But it seems the persistent mode work. Would we be ok with having only one k8s test for a while?

[airflow] branch master updated: Add query count test for LocalTaskJob (#8922)

2020-05-27 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 369e637 Add query count test for

[GitHub] [airflow] turbaszek merged pull request #8922: Add query count test for LocalTaskJob

2020-05-27 Thread GitBox
turbaszek merged pull request #8922: URL: https://github.com/apache/airflow/pull/8922 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431580206 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 =

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431580206 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 =

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431578845 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] aneesh-joseph opened a new pull request #9044: do a kinit before yarn kill if keytab and principal is provided

2020-05-27 Thread GitBox
aneesh-joseph opened a new pull request #9044: URL: https://github.com/apache/airflow/pull/9044 --- When the spark submit hook's `on_kill` is called, it tries to kill the yarn app, but doesn't do a kinit even if the keytab and principal is supplied. This would mean that the `yarn

[GitHub] [airflow] msumit commented on a change in pull request #8265: Kubernetes on host

2020-05-27 Thread GitBox
msumit commented on a change in pull request #8265: URL: https://github.com/apache/airflow/pull/8265#discussion_r431569992 ## File path: scripts/ci/kubernetes/app/templates/airflow.template.yaml ## @@ -42,33 +42,43 @@ spec: name: airflow spec:

[airflow] tag nightly-master updated (0b0e4f7 -> 7386670)

2020-05-27 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 0b0e4f7 (commit) to 7386670 (commit) from

[jira] [Commented] (AIRFLOW-6395) [AIP-28] Add AsyncExecutor option

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118237#comment-17118237 ] ASF GitHub Bot commented on AIRFLOW-6395: - dazza-codes commented on pull request #6984: URL:

[GitHub] [airflow] dazza-codes commented on pull request #6984: [WIP][AIRFLOW-6395] Add an asyncio compatible dask executor [AIP-28]

2020-05-27 Thread GitBox
dazza-codes commented on pull request #6984: URL: https://github.com/apache/airflow/pull/6984#issuecomment-635054207 For various reasons beyond my control, the complexity of Airflow seemed to be a hurdle for various ops and other personnel on a team at work. In short, I don't have time

[GitHub] [airflow] houqp commented on pull request #9042: detect incompatible docker server version in breeze

2020-05-27 Thread GitBox
houqp commented on pull request #9042: URL: https://github.com/apache/airflow/pull/9042#issuecomment-635027681 @potiuk I noticed travis build is already failing with this check: ``` Client: Docker Engine - Community Version: 19.03.9 API version: 1.40 Go

[GitHub] [airflow] ephraimbuddy opened a new pull request #9043: Add example dag and system test for LocalFilesystemToGCSOperator

2020-05-27 Thread GitBox
ephraimbuddy opened a new pull request #9043: URL: https://github.com/apache/airflow/pull/9043 --- This PR closes one of the issues in #8280 Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests

[GitHub] [airflow] houqp opened a new pull request #9042: detect incompatible docker server version in breeze

2020-05-27 Thread GitBox
houqp opened a new pull request #9042: URL: https://github.com/apache/airflow/pull/9042 Without this detection code, breeze just fails with the following error: ``` ===

[GitHub] [airflow] casassg commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431484377 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 = DummyOperator(task_id="op4",

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431476613 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag

[GitHub] [airflow] casassg opened a new issue #9041: [AIP-31] Update Airflow tutorial to use functional DAGs

2020-05-27 Thread GitBox
casassg opened a new issue #9041: URL: https://github.com/apache/airflow/issues/9041 **Description** Change Airflow tutorial (or create new tutorial) using the new functional layer. **Use case / motivation** Help users onboard Airflow by reducing the complexity of

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431464263 ## File path: tests/operators/test_python.py ## @@ -311,6 +315,350 @@ def func(**context): python_operator.run(start_date=DEFAULT_DATE,

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431452493 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9040: Correctly describe the dst parameter in LocalFilesystemToGCS operator

2020-05-27 Thread GitBox
boring-cyborg[bot] commented on issue #9040: URL: https://github.com/apache/airflow/issues/9040#issuecomment-634950607 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431452493 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] ephraimbuddy opened a new issue #9040: Correctly describe the dst parameter in LocalFilesystemToGCS operator

2020-05-27 Thread GitBox
ephraimbuddy opened a new issue #9040: URL: https://github.com/apache/airflow/issues/9040 **Apache Airflow version**: 2.0.0 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431450538 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431450538 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[jira] [Commented] (AIRFLOW-5500) Bug in trigger api endpoint

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118034#comment-17118034 ] ASF GitHub Bot commented on AIRFLOW-5500: - kylegao91 commented on pull request #8081: URL:

[GitHub] [airflow] kylegao91 commented on pull request #8081: [AIRFLOW-5500] Fix the trigger_dag api in the case of nested subdags

2020-05-27 Thread GitBox
kylegao91 commented on pull request #8081: URL: https://github.com/apache/airflow/pull/8081#issuecomment-634905164 Really hope this can be merged into 1.10.11, very important for our tasks. This is an automated message from

[GitHub] [airflow] dimberman commented on a change in pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
dimberman commented on a change in pull request #9038: URL: https://github.com/apache/airflow/pull/9038#discussion_r431398123 ## File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh ## @@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/ echo echo

[GitHub] [airflow] potiuk commented on a change in pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
potiuk commented on a change in pull request #9038: URL: https://github.com/apache/airflow/pull/9038#discussion_r431367442 ## File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh ## @@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/ echo echo

[GitHub] [airflow] feluelle commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
feluelle commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431366969 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag

[GitHub] [airflow] potiuk commented on issue #8712: docker image does not include 'pymssql'

2020-05-27 Thread GitBox
potiuk commented on issue #8712: URL: https://github.com/apache/airflow/issues/8712#issuecomment-634864391 Yep. We want to minimise the prod image size to get the most frequently used packages (and core ones). We've added a new feature today to the prod image - you can add additional

[GitHub] [airflow] potiuk commented on pull request #8976: You can push with Breeze as separate command and to cache

2020-05-27 Thread GitBox
potiuk commented on pull request #8976: URL: https://github.com/apache/airflow/pull/8976#issuecomment-634859654 @ashb @dimberman @kaxil @mik-laj -> that one wil make it easier to iterate on CI/Prod images - as I will be switching back to it, It would be useful to merge it.

[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117987#comment-17117987 ] ASF GitHub Bot commented on AIRFLOW-3964: - YingboWang commented on pull request #5499: URL:

[GitHub] [airflow] YingboWang commented on pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor

2020-05-27 Thread GitBox
YingboWang commented on pull request #5499: URL: https://github.com/apache/airflow/pull/5499#issuecomment-634854013 Thanks @KevinYang21 for the comment. This PR description was updated with a high level architecture diagram and some charts for smart sensor impact after being deployed.

[GitHub] [airflow] alexbegg commented on issue #8712: docker image does not include 'pymssql'

2020-05-27 Thread GitBox
alexbegg commented on issue #8712: URL: https://github.com/apache/airflow/issues/8712#issuecomment-634841172 pymssql is installed with the airflow extra "mssql". It is not pre-installed with the docker image apache/airflow:1.10.10. I am currently installing it as a separate pip install of

[GitHub] [airflow] potiuk commented on pull request #9039: Updated missing parameters for docker image building

2020-05-27 Thread GitBox
potiuk commented on pull request #9039: URL: https://github.com/apache/airflow/pull/9039#issuecomment-634834608 cc @wittfabian This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk opened a new pull request #9039: Updated missing parameters for docker image building

2020-05-27 Thread GitBox
potiuk opened a new pull request #9039: URL: https://github.com/apache/airflow/pull/9039 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] potiuk commented on pull request #8974: Vault has now VaultHook not only SecretBackend

2020-05-27 Thread GitBox
potiuk commented on pull request #8974: URL: https://github.com/apache/airflow/pull/8974#issuecomment-634822700 Hey @kaxil -> I split the tests into separate client_vault (more detailed) and hook_vault tests (focused on "green-path" + reading parameters from connections). I think this way

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431318258 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431313499 ## File path: airflow/models/dag.py ## @@ -1468,14 +1472,26 @@ def create_dagrun(self, :param session: database session :type session:

[GitHub] [airflow] turbaszek commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-05-27 Thread GitBox
turbaszek commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-634816078 Summoning @edejong to hear his opinion :) This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] turbaszek commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431311590 ## File path: airflow/models/dag.py ## @@ -1468,14 +1472,26 @@ def create_dagrun(self, :param session: database session :type

[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431310411 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431309657 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431308938 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[GitHub] [airflow] dimberman commented on a change in pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
dimberman commented on a change in pull request #9038: URL: https://github.com/apache/airflow/pull/9038#discussion_r431297732 ## File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh ## @@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/ echo echo

[GitHub] [airflow] mdediana commented on issue #8649: Add support for more than 1 cron exp per DAG

2020-05-27 Thread GitBox
mdediana commented on issue #8649: URL: https://github.com/apache/airflow/issues/8649#issuecomment-634780827 @mik-laj Sure, I will do that, thanks. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] themantalope commented on issue #8649: Add support for more than 1 cron exp per DAG

2020-05-27 Thread GitBox
themantalope commented on issue #8649: URL: https://github.com/apache/airflow/issues/8649#issuecomment-634766517 @mik-laj I would recommend that the user be allow to supply a list of cron strings or cron strings with comma separation. I would then implement a object that has

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431247683 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] samuelkhtu commented on pull request #8942: #8525 Add SQL Branch Operator

2020-05-27 Thread GitBox
samuelkhtu commented on pull request #8942: URL: https://github.com/apache/airflow/pull/8942#issuecomment-634753579 > @samuelkhtu we can do it in a separate PR. This will allow us to build a better git history. Thanks @mik-laj and @eladkal , can someone help and approve this PR? I

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431236903 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431236903 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] optimuspaul commented on issue #8281: CLI trigger_dag with json_client enabled raises TypeError: Object of type Pendulum is not JSON serializable

2020-05-27 Thread GitBox
optimuspaul commented on issue #8281: URL: https://github.com/apache/airflow/issues/8281#issuecomment-634746271 https://github.com/apache/airflow/pull/8384 This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431236903 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431236903 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431234943 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431235391 ## File path: openapi.yaml ## @@ -0,0 +1,2411 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431234581 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431234192 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431233852 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431230679 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[airflow] branch master updated (8ac90b0 -> 7386670)

2020-05-27 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 8ac90b0 [AIRFLOW-5615] Reduce duplicated logic around job heartbeating (#6311) add 7386670 Additional

[GitHub] [airflow] potiuk merged pull request #9035: Additional python extras and deps can be set in breeze

2020-05-27 Thread GitBox
potiuk merged pull request #9035: URL: https://github.com/apache/airflow/pull/9035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk closed issue #8604: Add ADDITIONAL_PYTHON_DEPS to the image

2020-05-27 Thread GitBox
potiuk closed issue #8604: URL: https://github.com/apache/airflow/issues/8604 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk closed issue #8866: Additional extras

2020-05-27 Thread GitBox
potiuk closed issue #8866: URL: https://github.com/apache/airflow/issues/8866 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431205011 ## File path: openapi.yaml ## @@ -0,0 +1,2411 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431196294 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] ashb commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
ashb commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431190655 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431176549 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431176549 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431187909 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] mik-laj commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431176549 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] joppevos opened a new pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
joppevos opened a new pull request #9037: URL: https://github.com/apache/airflow/pull/9037 A simple guide based on the example_dataproc file. Addressing this [issue](https://github.com/apache/airflow/issues/8203) # qs - @mik-laj All the examples we have of job configurations

[GitHub] [airflow] mik-laj commented on a change in pull request #9030: Allow using Airflow with Flask CLI

2020-05-27 Thread GitBox
mik-laj commented on a change in pull request #9030: URL: https://github.com/apache/airflow/pull/9030#discussion_r431166489 ## File path: airflow/www/app.py ## @@ -70,6 +73,31 @@ def create_app(config=None, testing=False, app_name="Airflow"): app.json_encoder =

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9036: Task instance log_filepath doesn't include try_number

2020-05-27 Thread GitBox
boring-cyborg[bot] commented on issue #9036: URL: https://github.com/apache/airflow/issues/9036#issuecomment-634671177 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] Pymancer opened a new issue #9036: Task instance log_filepath doesn't include try_number

2020-05-27 Thread GitBox
Pymancer opened a new issue #9036: URL: https://github.com/apache/airflow/issues/9036 Is it correct that ti.log_filepath doesn't include try_number? How to get a valid path with it?

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431140537 ## File path: airflow/models/dagrun.py ## @@ -54,25 +54,27 @@ class DagRun(Base, LoggingMixin): _state = Column('state', String(50),

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431137238 ## File path: airflow/models/dag.py ## @@ -1468,14 +1472,26 @@ def create_dagrun(self, :param session: database session :type session:

[GitHub] [airflow] potiuk opened a new pull request #9035: Additional python extras and deps can be set in breeze

2020-05-27 Thread GitBox
potiuk opened a new pull request #9035: URL: https://github.com/apache/airflow/pull/9035 Closes #8604 Closes #8866 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for

[GitHub] [airflow] potiuk commented on pull request #9035: Additional python extras and deps can be set in breeze

2020-05-27 Thread GitBox
potiuk commented on pull request #9035: URL: https://github.com/apache/airflow/pull/9035#issuecomment-634665722 CC @wittfabian This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk commented on issue #8872: Add dependencies on build image

2020-05-27 Thread GitBox
potiuk commented on issue #8872: URL: https://github.com/apache/airflow/issues/8872#issuecomment-634664427 Yeah, I see the point. I think also maybe we should not do onbuild if we can add easily build args to supplement that.

[GitHub] [airflow] potiuk commented on issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
potiuk commented on issue #9033: URL: https://github.com/apache/airflow/issues/9033#issuecomment-634630189 For 1.10.10 it was like that but it's been fixed in master and will be cherry-picked for 1.10.11 This is an

[GitHub] [airflow] potiuk closed issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
potiuk closed issue #9033: URL: https://github.com/apache/airflow/issues/9033 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] zikun commented on issue #8872: Add dependencies on build image

2020-05-27 Thread GitBox
zikun commented on issue #8872: URL: https://github.com/apache/airflow/issues/8872#issuecomment-634616072 This could be a nice feature but looks like there was an effort to remove `ONBUILD` from official images - https://github.com/docker-library/official-images/issues/2076 As the

[GitHub] [airflow] feluelle commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
feluelle commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431035377 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args,

[GitHub] [airflow] mik-laj commented on issue #9034: Fix GoogleDiscoveryApiHook docs warnings

2020-05-27 Thread GitBox
mik-laj commented on issue #9034: URL: https://github.com/apache/airflow/issues/9034#issuecomment-634606513 Can you remove _api and _build directory? That should solve this problem. This is an automated message from the

[GitHub] [airflow-site] mschickensoup commented on pull request #268: adding a logo for sift use case

2020-05-27 Thread GitBox
mschickensoup commented on pull request #268: URL: https://github.com/apache/airflow-site/pull/268#issuecomment-634605066 I should be available to take a look at it around Friday, would it work? @mik-laj This is an

[GitHub] [airflow] turbaszek opened a new issue #9034: Fix GoogleDiscoveryApiHook docs warnings

2020-05-27 Thread GitBox
turbaszek opened a new issue #9034: URL: https://github.com/apache/airflow/issues/9034 **Apache Airflow version**: master branch **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A **Environment**: - breeze **What happened**:

[GitHub] [airflow] zikun edited a comment on issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
zikun edited a comment on issue #9033: URL: https://github.com/apache/airflow/issues/9033#issuecomment-634600902 Before I make a PR to fix it, can someone confirm entering bash shell is the expected behaviour? This is an

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
boring-cyborg[bot] commented on issue #9033: URL: https://github.com/apache/airflow/issues/9033#issuecomment-634600571 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] zikun commented on issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
zikun commented on issue #9033: URL: https://github.com/apache/airflow/issues/9033#issuecomment-634600902 Before I make a PR to fix it, can someone confirm this is the expected behaviour? This is an automated message from

[GitHub] [airflow] zikun opened a new issue #9033: entrypoint cannot enter bash shell

2020-05-27 Thread GitBox
zikun opened a new issue #9033: URL: https://github.com/apache/airflow/issues/9033 **Apache Airflow version**: 1.10.10 / master **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: Production Docker image

[GitHub] [airflow] mik-laj commented on pull request #9023: Add snowflake to slack operator

2020-05-27 Thread GitBox
mik-laj commented on pull request #9023: URL: https://github.com/apache/airflow/pull/9023#issuecomment-634600193 @ad-m We have temporary problems with Kubernetes tests running on Travis Ci. This change does not apply to Kubernetes, so it is safe. @dimberman is working on improvements for

[GitHub] [airflow] kaxil commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431046930 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow-site] mik-laj commented on pull request #268: adding a logo for sift use case

2020-05-27 Thread GitBox
mik-laj commented on pull request #268: URL: https://github.com/apache/airflow-site/pull/268#issuecomment-634597352 @mschickensoup Do you have time to look at it? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] stijndehaes commented on issue #8963: SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version exception

2020-05-27 Thread GitBox
stijndehaes commented on issue #8963: URL: https://github.com/apache/airflow/issues/8963#issuecomment-634596428 @ywan2017 I also have a PR open on airflow to work with spark 3.0 https://github.com/apache/airflow/pull/8730

[GitHub] [airflow] BasPH commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
BasPH commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431044194 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] BasPH commented on a change in pull request #8721: Add OpenAPI specification (II)

2020-05-27 Thread GitBox
BasPH commented on a change in pull request #8721: URL: https://github.com/apache/airflow/pull/8721#discussion_r431044194 ## File path: openapi.yaml ## @@ -0,0 +1,2427 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic

2020-05-27 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117664#comment-17117664 ] ASF subversion and git services commented on AIRFLOW-5615: -- Commit

[airflow] branch master updated (30b12a9 -> 8ac90b0)

2020-05-27 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 30b12a9 Filter dags by clicking on tag (#8897) add 8ac90b0 [AIRFLOW-5615] Reduce duplicated logic around job

[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117663#comment-17117663 ] ASF GitHub Bot commented on AIRFLOW-5615: - ashb merged pull request #6311: URL:

  1   2   >