[GitHub] [airflow] potiuk commented on pull request #8918: Impala hook implention

2020-07-16 Thread GitBox
potiuk commented on pull request #8918: URL: https://github.com/apache/airflow/pull/8918#issuecomment-659889798 I guess we will have to wait with the merge unti impyla releases next version I am afraid. This is an automated

[GitHub] [airflow] potiuk commented on pull request #8918: Impala hook implention

2020-07-16 Thread GitBox
potiuk commented on pull request #8918: URL: https://github.com/apache/airflow/pull/8918#issuecomment-659890063 Sorry for that :( This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [airflow] potiuk commented on pull request #8918: Impala hook implention

2020-07-16 Thread GitBox
potiuk commented on pull request #8918: URL: https://github.com/apache/airflow/pull/8918#issuecomment-659889219 That's an interesting one. It looks like there is a conflict of dependencies between Impala and Hive (because of incompatible versions of thrift used - the root cause is o

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9864: Allow localization of UI

2020-07-16 Thread GitBox
boring-cyborg[bot] commented on issue #9864: URL: https://github.com/apache/airflow/issues/9864#issuecomment-659873307 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] yarons opened a new issue #9864: Allow localization of UI

2020-07-16 Thread GitBox
yarons opened a new issue #9864: URL: https://github.com/apache/airflow/issues/9864 **Description** I want to be able to translate the UI and possibly also adapt it to RTL (for Hebrew, Arabic, Persian, etc.) **Use case / motivation** I think that Airflow is very friendl

[GitHub] [airflow] potiuk closed issue #9863: Resource requirement for Kubernetes Executor

2020-07-16 Thread GitBox
potiuk closed issue #9863: URL: https://github.com/apache/airflow/issues/9863 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [airflow] potiuk commented on issue #9863: Resource requirement for Kubernetes Executor

2020-07-16 Thread GitBox
potiuk commented on issue #9863: URL: https://github.com/apache/airflow/issues/9863#issuecomment-659860904 Hello @sabir-piludiya - as explained in the template please use our slack #troubleshooting channel in slack as described in https://airflow.apache.org/community/ ---

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9863: Resource requirement for Kubernetes Executor

2020-07-16 Thread GitBox
boring-cyborg[bot] commented on issue #9863: URL: https://github.com/apache/airflow/issues/9863#issuecomment-659855663 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] sabir-piludiya opened a new issue #9863: Resource requirement for Kubernetes Executor

2020-07-16 Thread GitBox
sabir-piludiya opened a new issue #9863: URL: https://github.com/apache/airflow/issues/9863 I would like to know how much resource is required to setup Airflow in Kubernetes and having Kubernetes executor. I haven't find the required memory and CPU for that anywhere.

[GitHub] [airflow] uns1 commented on issue #9852: dag_run.conf is NoneType when running example_trigger_target_dag from example_dags

2020-07-16 Thread GitBox
uns1 commented on issue #9852: URL: https://github.com/apache/airflow/issues/9852#issuecomment-659830584 Thanks @kaxil! While this ensures that conf will always return a dictionary and not None, my issue is that conf is not being passed from the controller to the target dag at all - as per

[GitHub] [airflow] uns1 commented on pull request #9853: Fix DagRun.conf when using trigger_dag API

2020-07-16 Thread GitBox
uns1 commented on pull request #9853: URL: https://github.com/apache/airflow/pull/9853#issuecomment-659820987 Thanks @kaxil. While this fixes the fact that the returned object is always a dictionary, my issue was that the conf parameter isn't getting passed to the target dag. E.g.,

[GitHub] [airflow] liuandy commented on a change in pull request #9843: Add Snowflake support to SQL operator and sensor

2020-07-16 Thread GitBox
liuandy commented on a change in pull request #9843: URL: https://github.com/apache/airflow/pull/9843#discussion_r456192285 ## File path: tests/models/test_connection.py ## @@ -533,3 +534,7 @@ def test_enforce_alphabetical_order(self): expected_keys = sorted(current_ke

[GitHub] [airflow] tanjinP commented on pull request #9079: [8970] Improve KubernetesPodOperator guide

2020-07-16 Thread GitBox
tanjinP commented on pull request #9079: URL: https://github.com/apache/airflow/pull/9079#issuecomment-659774270 @mik-laj > I would be happy to add information that Kubernetes Executor is not required for Kubernetes Pod Operator. Mentioned this as a [note here](https://github.c

[GitHub] [airflow] kaxil opened a new pull request #9862: Add __repr__ to SerializedDagModel

2020-07-16 Thread GitBox
kaxil opened a new pull request #9862: URL: https://github.com/apache/airflow/pull/9862 Before: `` After: `` --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not

[GitHub] [airflow] mik-laj commented on a change in pull request #9861: [WIP] Add test for GCSTaskHandler (#9600)

2020-07-16 Thread GitBox
mik-laj commented on a change in pull request #9861: URL: https://github.com/apache/airflow/pull/9861#discussion_r456127350 ## File path: airflow/providers/google/cloud/log/gcs_task_handler.py ## @@ -55,8 +55,8 @@ def hook(self): except Exception as e: # pylint: disab

[GitHub] [airflow] mik-laj commented on a change in pull request #9861: [WIP] Add test for GCSTaskHandler (#9600)

2020-07-16 Thread GitBox
mik-laj commented on a change in pull request #9861: URL: https://github.com/apache/airflow/pull/9861#discussion_r456127240 ## File path: airflow/providers/google/cloud/log/gcs_task_handler.py ## @@ -165,7 +165,7 @@ def gcs_write(self, log, remote_log_location, append=True):

[GitHub] [airflow] odracci commented on pull request #9845: Fix PodGenerator to handle Kubernetes resources

2020-07-16 Thread GitBox
odracci commented on pull request #9845: URL: https://github.com/apache/airflow/pull/9845#issuecomment-659699854 The build fails with: `Could not find a version that satisfies the requirement pandas==0.25.3 (from -c https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirement

[GitHub] [airflow] OmairK opened a new pull request #9861: Add test for GCSTaskHandler (#9600)

2020-07-16 Thread GitBox
OmairK opened a new pull request #9861: URL: https://github.com/apache/airflow/pull/9861 Fix #9600 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for docum

[GitHub] [airflow] BrunoDamacena closed issue #8160: SimpleHttpOperator aborts connection after 5 minutes

2020-07-16 Thread GitBox
BrunoDamacena closed issue #8160: URL: https://github.com/apache/airflow/issues/8160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk commented on pull request #9654: Spllit ci workflow

2020-07-16 Thread GitBox
potiuk commented on pull request #9654: URL: https://github.com/apache/airflow/pull/9654#issuecomment-659644839 @kaxil @ashb @turbaszek @mik-laj . Here it is - I split the Ci workflow in smaller ones. Each workflow is independent and can be rerun separately as needed. cancel steps are inc

[GitHub] [airflow] blcksrx commented on pull request #8918: Impala hook implention

2020-07-16 Thread GitBox
blcksrx commented on pull request #8918: URL: https://github.com/apache/airflow/pull/8918#issuecomment-659627891 @potiuk I fixed the static checks! but currently, there are issues related to `pyhive`: ``` AttributeError: module 'pyhive' has no attribute 'hive' ``` -

[GitHub] [airflow] YingboWang commented on pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor

2020-07-16 Thread GitBox
YingboWang commented on pull request #5499: URL: https://github.com/apache/airflow/pull/5499#issuecomment-659618394 @ashb Thank you for your comments in AIP-17. I agree that it's a good idea to leverage the smart sensor for more operators. I'm just wondering if you can help take a look int

[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks

2020-07-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159412#comment-17159412 ] ASF GitHub Bot commented on AIRFLOW-3964: - YingboWang commented on pull request

[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator

2020-07-16 Thread GitBox
subkanthi commented on issue #9134: URL: https://github.com/apache/airflow/issues/9134#issuecomment-659608175 @mik-laj , @zhongjiajie is it OK if i just work on the simple flinksubmit operator first, maybe we can have another issue for the streaming part?

[GitHub] [airflow] janarthanandev commented on pull request #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor

2020-07-16 Thread GitBox
janarthanandev commented on pull request #7163: URL: https://github.com/apache/airflow/pull/7163#issuecomment-659601976 Great work @roitvt This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Commented] (AIRFLOW-6542) sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

2020-07-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159405#comment-17159405 ] ASF GitHub Bot commented on AIRFLOW-6542: - janarthanandev commented on pull req

[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks

2020-07-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159393#comment-17159393 ] ASF GitHub Bot commented on AIRFLOW-3964: - YingboWang commented on pull request

[GitHub] [airflow] morrme commented on pull request #9775: improve typing for datadog provider

2020-07-16 Thread GitBox
morrme commented on pull request #9775: URL: https://github.com/apache/airflow/pull/9775#issuecomment-659588398 @OmairK @potiuk This should be ready for review :) - This is an automated message from the Apache Git Service. To

[GitHub] [airflow] YingboWang commented on pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor

2020-07-16 Thread GitBox
YingboWang commented on pull request #5499: URL: https://github.com/apache/airflow/pull/5499#issuecomment-659587164 @BasPH Thank you for giving us feedback in an early stage. We have done a lot of work on this smart sensor project since then. Would you like to take a look at this PR again?

[GitHub] [airflow] potiuk commented on a change in pull request #9841: Add log of affected sql rows with Cursor.rowcount

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9841: URL: https://github.com/apache/airflow/pull/9841#discussion_r455982811 ## File path: tests/providers/postgres/hooks/test_postgres.py ## @@ -258,3 +258,16 @@ def test_insert_rows_replace_missing_replace_index_arg(self):

[GitHub] [airflow] nitaibezerra commented on pull request #9770: Add ME-Br to who uses Airflow list

2020-07-16 Thread GitBox
nitaibezerra commented on pull request #9770: URL: https://github.com/apache/airflow/pull/9770#issuecomment-659549991 Thanks @potiuk for accepting this addition. We've been successfully using Airflow for the last 8 months. I watched the Summit talk you've mentioned and I was appalled by th

[GitHub] [airflow] kaxil commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455937862 ## File path: tests/models/test_dagbag.py ## @@ -657,3 +659,35 @@ def test_serialized_dags_are_written_to_db_on_sync(self): new_serialized_da

[GitHub] [airflow] kaxil commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455937662 ## File path: tests/models/test_dagbag.py ## @@ -657,3 +659,35 @@ def test_serialized_dags_are_written_to_db_on_sync(self): new_serialized_da

[GitHub] [airflow] kaxil commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455934821 ## File path: airflow/models/serialized_dag.py ## @@ -233,3 +233,18 @@ def bulk_sync_to_db(dags: List[DAG], session: Session = None):

[GitHub] [airflow] olchas edited a comment on issue #8803: Impersonate service accounts while running GCP Operators without key material (if airflow is running on GCP)

2020-07-16 Thread GitBox
olchas edited a comment on issue #8803: URL: https://github.com/apache/airflow/issues/8803#issuecomment-659540464 @mik-laj sure. Could you assign me to this issue, please? @ad-m, thanks for the offer. I am uncertain about supporting impersonation via environment variable. As far as I

[GitHub] [airflow] olchas commented on issue #8803: Impersonate service accounts while running GCP Operators without key material (if airflow is running on GCP)

2020-07-16 Thread GitBox
olchas commented on issue #8803: URL: https://github.com/apache/airflow/issues/8803#issuecomment-659540464 @mik-laj sure. Could you assign me to this issue, please? @ad-m, thanks for the offer. I am uncertain about supporting impersonation via environment variable. As far as I know,

[GitHub] [airflow] marcusianlevine opened a new issue #9860: Kubernetes Executor Config Volumes Break Airflow UI

2020-07-16 Thread GitBox
marcusianlevine opened a new issue #9860: URL: https://github.com/apache/airflow/issues/9860 **Apache Airflow version**: 1.10.11 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.16.9 **Environment**: - **Cloud provider or hardware configu

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9860: Kubernetes Executor Config Volumes Break Airflow UI

2020-07-16 Thread GitBox
boring-cyborg[bot] commented on issue #9860: URL: https://github.com/apache/airflow/issues/9860#issuecomment-659539902 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] cjsekl commented on a change in pull request #9841: Add log of affected sql rows with Cursor.rowcount

2020-07-16 Thread GitBox
cjsekl commented on a change in pull request #9841: URL: https://github.com/apache/airflow/pull/9841#discussion_r455903989 ## File path: tests/providers/postgres/hooks/test_postgres.py ## @@ -258,3 +258,16 @@ def test_insert_rows_replace_missing_replace_index_arg(self):

[airflow] branch master updated (473f506 -> faec41e)

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 473f506 Add imagePullSecrets to the create user job (#9802) add faec41e Group CI scripts in subdirectories (#

[GitHub] [airflow] potiuk merged pull request #9653: Group CI scripts in subdirectories

2020-07-16 Thread GitBox
potiuk merged pull request #9653: URL: https://github.com/apache/airflow/pull/9653 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] cjsekl commented on a change in pull request #9841: Add log of affected sql rows with Cursor.rowcount

2020-07-16 Thread GitBox
cjsekl commented on a change in pull request #9841: URL: https://github.com/apache/airflow/pull/9841#discussion_r455896438 ## File path: tests/providers/postgres/hooks/test_postgres.py ## @@ -258,3 +258,16 @@ def test_insert_rows_replace_missing_replace_index_arg(self):

[GitHub] [airflow] cjsekl commented on a change in pull request #9841: Add log of affected sql rows with Cursor.rowcount

2020-07-16 Thread GitBox
cjsekl commented on a change in pull request #9841: URL: https://github.com/apache/airflow/pull/9841#discussion_r455895095 ## File path: tests/providers/postgres/hooks/test_postgres.py ## @@ -258,3 +258,16 @@ def test_insert_rows_replace_missing_replace_index_arg(self):

[GitHub] [airflow] kaxil commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455851227 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[airflow] 01/01: Fixes sqlalchemy flask failure with python 3.8.4

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit f1fba28a7d5cac9b878483e2a32a8ad490a1035a Author: Jarek Potiuk AuthorDate: Tue Jul 14 19:29:13 2020 +0200 Fixes

[airflow] branch v1-10-test updated (235ab10 -> f1fba28)

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 235ab10 Fixes sqlalchemy flask failure with python 3.8.4 new f1fba28 Fixes sqlalchemy flask failure with

[airflow] branch v1-10-test updated (e983da7 -> 235ab10)

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit e983da7 Fixes sqlalchemy flask failure with python 3.8.4 new 235ab10 Fixes sqlalchemy flask failure with

[airflow] 01/01: Fixes sqlalchemy flask failure with python 3.8.4

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 235ab1045f0bb074925c5b80fe2e1513f6e8d86b Author: Jarek Potiuk AuthorDate: Tue Jul 14 19:29:13 2020 +0200 Fixes

[GitHub] [airflow] potiuk commented on pull request #9845: Fix PodGenerator to handle Kubernetes resources

2020-07-16 Thread GitBox
potiuk commented on pull request #9845: URL: https://github.com/apache/airflow/pull/9845#issuecomment-659450887 Just line too long for flake @kaxil :( This is an automated message from the Apache Git Service. To respond to th

[GitHub] [airflow] potiuk closed issue #9801: Helm create default user job unable to pull private registry image

2020-07-16 Thread GitBox
potiuk closed issue #9801: URL: https://github.com/apache/airflow/issues/9801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[airflow] branch master updated (f4067b6 -> 473f506)

2020-07-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from f4067b6 Fix Experimental API Client (#9849) add 473f506 Add imagePullSecrets to the create user job (#9802)

[GitHub] [airflow] potiuk merged pull request #9802: Add imagePullSecrets to the create user job

2020-07-16 Thread GitBox
potiuk merged pull request #9802: URL: https://github.com/apache/airflow/pull/9802 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] potiuk commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg

2020-07-16 Thread GitBox
potiuk commented on pull request #9816: URL: https://github.com/apache/airflow/pull/9816#issuecomment-659423399 I think it depends if we want to merge it to 1.10 branch at all and whether we release helm chart from master or v1-10-test/stable. If we add more tests and finalize #9663 then w

[jira] [Commented] (AIRFLOW-46) Running Trigger DAG test with SequentialExecutor yields error

2020-07-16 Thread U Saifi (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-46?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159223#comment-17159223 ] U Saifi commented on AIRFLOW-46: I'm experiencing the exact same thing on Airflow 1.10.11

[GitHub] [airflow] mik-laj merged pull request #9849: Fix Experimental API Client

2020-07-16 Thread GitBox
mik-laj merged pull request #9849: URL: https://github.com/apache/airflow/pull/9849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[airflow] branch master updated (eb6f1d1 -> f4067b6)

2020-07-16 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from eb6f1d1 Fix typo in datafusion operator (#9859) add f4067b6 Fix Experimental API Client (#9849) No new

[airflow] branch master updated (e4790d5 -> eb6f1d1)

2020-07-16 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from e4790d5 Helm chart can now place arbitrary config settings in to airflow.cfg (#9816) add eb6f1d1 Fix typo

[GitHub] [airflow] turbaszek merged pull request #9859: Fixed typo in datafusion operator

2020-07-16 Thread GitBox
turbaszek merged pull request #9859: URL: https://github.com/apache/airflow/pull/9859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [airflow] ashb commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg

2020-07-16 Thread GitBox
ashb commented on pull request #9816: URL: https://github.com/apache/airflow/pull/9816#issuecomment-659415083 Should we target 1.10.12 for this? This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] ashb merged pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg

2020-07-16 Thread GitBox
ashb merged pull request #9816: URL: https://github.com/apache/airflow/pull/9816 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[airflow] branch master updated (31cab8f -> e4790d5)

2020-07-16 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 31cab8f Fix DagRun.conf when using trigger_dag API (#9853) add e4790d5 Helm chart can now place arbitrary config

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455785037 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] michalslowikowski00 opened a new pull request #9859: Fixed typo in datafusion operator

2020-07-16 Thread GitBox
michalslowikowski00 opened a new pull request #9859: URL: https://github.com/apache/airflow/pull/9859 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for docume

[GitHub] [airflow] aneesh-joseph commented on pull request #9371: Add git sync option and unit tests for the Helm chart

2020-07-16 Thread GitBox
aneesh-joseph commented on pull request #9371: URL: https://github.com/apache/airflow/pull/9371#issuecomment-659411990 > I got an issue yesterday with git-sync and the K8s worker pods. > If we specify: > AIRFLOW__KUBERNETES__RUN_AS_USER="5" in values, git-sync ends up with the fol

[GitHub] [airflow] olchas commented on issue #9461: Unclear documentation for the delegate_to parameter

2020-07-16 Thread GitBox
olchas commented on issue #9461: URL: https://github.com/apache/airflow/issues/9461#issuecomment-659411579 @mik-laj sure. Could you assign me to this issue, please? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455784778 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455785037 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] kaxil commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455783821 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455781724 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] scrambldchannel commented on pull request #9858: Add "all" to list of valid breeze integrations

2020-07-16 Thread GitBox
scrambldchannel commented on pull request #9858: URL: https://github.com/apache/airflow/pull/9858#issuecomment-659408779 > Something like that: > > ``` > if [[ "${AVAILABLE_INTEGRATIONS} all" != "${_BREEZE_ALLOWED_INTEGRATIONS}" ]]; then > ``` Ah ok, missed that. I'll ta

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455781724 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] potiuk commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455781724 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] potiuk commented on pull request #9847: Support for set in XCom serialization (fix #8703)

2020-07-16 Thread GitBox
potiuk commented on pull request #9847: URL: https://github.com/apache/airflow/pull/9847#issuecomment-659404289 I did not yet have time to take a deeper look - and a little busy today :(. I will take a look tomorrow I think! Nothing to be sorry about :). Tests are there to catch all those

[GitHub] [airflow] marclamberti commented on pull request #9371: Add git sync option and unit tests for the Helm chart

2020-07-16 Thread GitBox
marclamberti commented on pull request #9371: URL: https://github.com/apache/airflow/pull/9371#issuecomment-659400129 I got an issue yesterday with git-sync and the K8s worker pods. If we specify: AIRFLOW__KUBERNETES__RUN_AS_USER="5" in values, git-sync ends up with the following

[GitHub] [airflow] tatiana commented on pull request #9847: Support for set in XCom serialization (fix #8703)

2020-07-16 Thread GitBox
tatiana commented on pull request #9847: URL: https://github.com/apache/airflow/pull/9847#issuecomment-659389946 Thanks, @potiuk ! I'll solve those later today, sorry - I'll make sure the CI is happy. Do you have any comments on the implementation itself? Anything you'd like me to chang

[GitHub] [airflow] potiuk commented on pull request #9847: Support for set in XCom serialization (fix #8703)

2020-07-16 Thread GitBox
potiuk commented on pull request #9847: URL: https://github.com/apache/airflow/pull/9847#issuecomment-659387302 @tatiana There is a test failing which is likely related to XCom Serialization change.. This is an automated me

[GitHub] [airflow] kaxil commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455752565 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
feluelle commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455751413 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '{

[GitHub] [airflow] potiuk commented on pull request #9858: Add "all" to list of valid breeze integrations

2020-07-16 Thread GitBox
potiuk commented on pull request #9858: URL: https://github.com/apache/airflow/pull/9858#issuecomment-659381250 Something like that: ``` if [[ "${AVAILABLE_INTEGRATIONS} all" != "${_BREEZE_ALLOWED_INTEGRATIONS}" ]]; then ``` --

[GitHub] [airflow] potiuk commented on pull request #9858: Add "all" to list of valid breeze integrations

2020-07-16 Thread GitBox
potiuk commented on pull request #9858: URL: https://github.com/apache/airflow/pull/9858#issuecomment-659381090 You will also need to add "all" to pre_commit_check_integrations.sh and run `pre-commit run update-breeze-file` to update the docs. -

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
feluelle commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455750879 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '{

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
feluelle commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455748785 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '{

[GitHub] [airflow] scrambldchannel opened a new pull request #9858: Add "all" to list of valid breeze integrations

2020-07-16 Thread GitBox
scrambldchannel opened a new pull request #9858: URL: https://github.com/apache/airflow/pull/9858 Hi, I've tried to address https://github.com/apache/airflow/issues/9857. This change allows use of the "all" option. --- Make sure to mark the boxes below before creating PR:

[GitHub] [airflow] scrambldchannel opened a new issue #9857: Breeze - selecting "all" integrations option doesn't work

2020-07-16 Thread GitBox
scrambldchannel opened a new issue #9857: URL: https://github.com/apache/airflow/issues/9857 **Apache Airflow version**: 2.0.0.dev0 (ie master branch) **Kubernetes version: n/a **Environment**: - **Cloud provider or hardware configuration**: Local machine

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9857: Breeze - selecting "all" integrations option doesn't work

2020-07-16 Thread GitBox
boring-cyborg[bot] commented on issue #9857: URL: https://github.com/apache/airflow/issues/9857#issuecomment-659373154 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] potiuk commented on a change in pull request #9841: Add log of affected sql rows with Cursor.rowcount

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9841: URL: https://github.com/apache/airflow/pull/9841#discussion_r455740465 ## File path: tests/providers/postgres/hooks/test_postgres.py ## @@ -258,3 +258,16 @@ def test_insert_rows_replace_missing_replace_index_arg(self):

[GitHub] [airflow] turbaszek commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
turbaszek commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455737857 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '

[GitHub] [airflow] potiuk commented on a change in pull request #9631: Add function to get current context

2020-07-16 Thread GitBox
potiuk commented on a change in pull request #9631: URL: https://github.com/apache/airflow/pull/9631#discussion_r455737854 ## File path: airflow/models/taskinstance.py ## @@ -1132,7 +1133,8 @@ def signal_handler(signum, frame): # pylint: disable=unused-argument self.

[GitHub] [airflow] feluelle commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

2020-07-16 Thread GitBox
feluelle commented on issue #8688: URL: https://github.com/apache/airflow/issues/8688#issuecomment-659367130 Also: https://stackoverflow.com/questions/17687534/jinja2-variables-naming-are-variables-naming-restrictions-the-same-as-for-pyth I could not find an officiial source regardin

[GitHub] [airflow] feluelle edited a comment on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

2020-07-16 Thread GitBox
feluelle edited a comment on issue #8688: URL: https://github.com/apache/airflow/issues/8688#issuecomment-659367130 Also: https://stackoverflow.com/questions/17687534/jinja2-variables-naming-are-variables-naming-restrictions-the-same-as-for-pyth I could not find an official source re

[GitHub] [airflow] mik-laj opened a new issue #9856: Export connection/variable to file

2020-07-16 Thread GitBox
mik-laj opened a new issue #9856: URL: https://github.com/apache/airflow/issues/9856 Hello, We have a ticket about importing variables and variable https://github.com/apache/airflow/issues/9855 To facilitate the management, it would also be useful to create this file using the

[GitHub] [airflow] feluelle commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

2020-07-16 Thread GitBox
feluelle commented on issue #8688: URL: https://github.com/apache/airflow/issues/8688#issuecomment-659365603 Can you try `{{ var.json.get("csba-catalog").sandbox }}` and see if that works. Source: https://stackoverflow.com/questions/32744047/parsing-a-token-with-hyphen-in-jinja --

[GitHub] [airflow] mik-laj opened a new issue #9855: Import connections/variables from a file

2020-07-16 Thread GitBox
mik-laj opened a new issue #9855: URL: https://github.com/apache/airflow/issues/9855 Hello, We have a new local file system secret backend that allows us to load connections and variables from files. We support 3 file formats: YAML, JSON, ENV. https://airflow.readthedocs.io/en/l

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
feluelle commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455726402 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '{

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

2020-07-16 Thread GitBox
feluelle commented on a change in pull request #9394: URL: https://github.com/apache/airflow/pull/9394#discussion_r455724510 ## File path: tests/operators/test_python.py ## @@ -1222,23 +1230,42 @@ def f(templates_dict): self._run_as_operator(f, templates_dict={'ds': '{

[GitHub] [airflow] ashb commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
ashb commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455716361 ## File path: tests/models/test_dagbag.py ## @@ -657,3 +659,35 @@ def test_serialized_dags_are_written_to_db_on_sync(self): new_serialized_dag

[GitHub] [airflow] kaxil commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455716010 ## File path: tests/models/test_serialized_dag.py ## @@ -120,5 +150,5 @@ def test_bulk_sync_to_db(self): dags = [ DAG("dag_1"), DAG("d

[GitHub] [airflow] ashb commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
ashb commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455715554 ## File path: tests/models/test_dagbag.py ## @@ -657,3 +659,35 @@ def test_serialized_dags_are_written_to_db_on_sync(self): new_serialized_dag

[GitHub] [airflow] ashb commented on a change in pull request #9851: Update Serialized DAGs in Webserver when DAGs are Updated

2020-07-16 Thread GitBox
ashb commented on a change in pull request #9851: URL: https://github.com/apache/airflow/pull/9851#discussion_r455715190 ## File path: tests/models/test_dagbag.py ## @@ -657,3 +659,35 @@ def test_serialized_dags_are_written_to_db_on_sync(self): new_serialized_dag

[GitHub] [airflow] kaxil commented on a change in pull request #9850: Don't Update Serialized DAGs in DB if DAG didn't change

2020-07-16 Thread GitBox
kaxil commented on a change in pull request #9850: URL: https://github.com/apache/airflow/pull/9850#discussion_r455714688 ## File path: airflow/models/serialized_dag.py ## @@ -94,13 +97,26 @@ def write_dag(cls, dag: DAG, min_update_interval: Optional[int] = None, session=

  1   2   >