[GitHub] [airflow] bhavaniravi commented on issue #12282: Rewrite Breeze in Python

2022-02-10 Thread GitBox
bhavaniravi commented on issue #12282: URL: https://github.com/apache/airflow/issues/12282#issuecomment-1035954258 ✋ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] eladkal commented on issue #14919: HiveHook doesn't scale up to tables with more than 32,767 partitions

2022-02-10 Thread GitBox
eladkal commented on issue #14919: URL: https://github.com/apache/airflow/issues/14919#issuecomment-1035946210 @pingzh maybe you can help with that issue as you are familiar with partitions in hive -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] eladkal commented on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
eladkal commented on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035941640 @potiuk why the new GitHub provider is missing? https://github.com/apache/airflow/pull/21076 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] brcps12 commented on issue #21386: Pod Annotations does not be injected in workers when using KubernetesExecutor

2022-02-10 Thread GitBox
brcps12 commented on issue #21386: URL: https://github.com/apache/airflow/issues/21386#issuecomment-1035915506 Yes, if the `KubernetesExecutor` is used, it seems true that I can’t use the Airflow Worker Pod. However, if set the `executor` to `KubernetesExecutor`, I can specify the

[airflow] branch main updated (b6892c4 -> 833087f)

2022-02-10 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from b6892c4 Fix bug incorrectly removing action from role, rather than permission. (#21483) add 833087f Enable

[GitHub] [airflow] dstandish merged pull request #21385: Enable asynchronous job submission in BigQuery Insert Job

2022-02-10 Thread GitBox
dstandish merged pull request #21385: URL: https://github.com/apache/airflow/pull/21385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] sawaca96 commented on issue #21386: Pod Annotations does not be injected in workers when using KubernetesExecutor

2022-02-10 Thread GitBox
sawaca96 commented on issue #21386: URL: https://github.com/apache/airflow/issues/21386#issuecomment-1035901300 kubernetesExecutor can't use worker. isn't it? ``` Error: UPGRADE FAILED: execution error at (airflow/templates/_helpers/validate-values.tpl:43:5): If

[GitHub] [airflow] jhtimmins commented on pull request #21483: Fix bug incorrectly removing action from role, rather than permission.

2022-02-10 Thread GitBox
jhtimmins commented on pull request #21483: URL: https://github.com/apache/airflow/pull/21483#issuecomment-1035892923 To add some context about why this PR required modifying views.py: FAB maps a permission resource to a single view class. It assumes that the base actions defined on

[airflow] branch main updated (5dc6338 -> b6892c4)

2022-02-10 Thread jhtimmins
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 5dc6338 Chart: Fix elasticsearch URL when username/password are empty (#21222) add b6892c4 Fix bug

[GitHub] [airflow] jhtimmins merged pull request #21483: Fix bug incorrectly removing action from role, rather than permission.

2022-02-10 Thread GitBox
jhtimmins merged pull request #21483: URL: https://github.com/apache/airflow/pull/21483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch constraints-main updated: Updating constraints. Build id:1827617528

2022-02-10 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new cd87878 Updating

[GitHub] [airflow] jedcunningham commented on pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
jedcunningham commented on pull request #21501: URL: https://github.com/apache/airflow/pull/21501#issuecomment-1035881416 This might be a good place: https://airflow.apache.org/docs/apache-airflow/stable/faq.html?highlight=parsing#scheduling-dag-file-parsing -- This is an automated

[GitHub] [airflow] pingzh commented on pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
pingzh commented on pull request #21501: URL: https://github.com/apache/airflow/pull/21501#issuecomment-1035877332 > Oh, and we should add a small section in the docs somewhere too I'd think. @jedcunningham can you share the docs that I can put it? thanks -- This is an automated

[airflow] branch main updated (5a6a2d6 -> 5dc6338)

2022-02-10 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 5a6a2d6 add how-to guide for pig operator (#21498) add 5dc6338 Chart: Fix elasticsearch URL when

[GitHub] [airflow] jedcunningham merged pull request #21222: Helm: Fix elasticsearch URL in secret when username/password fields are empty,

2022-02-10 Thread GitBox
jedcunningham merged pull request #21222: URL: https://github.com/apache/airflow/pull/21222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham closed issue #20560: Elasticsearch connection user and password should be optional

2022-02-10 Thread GitBox
jedcunningham closed issue #20560: URL: https://github.com/apache/airflow/issues/20560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham commented on pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
jedcunningham commented on pull request #21501: URL: https://github.com/apache/airflow/pull/21501#issuecomment-1035871896 Oh, and we should add a small section in the docs somewhere too I'd think. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] jedcunningham commented on a change in pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #21501: URL: https://github.com/apache/airflow/pull/21501#discussion_r804348257 ## File path: tests/models/test_dagbag.py ## @@ -206,6 +207,49 @@ def test_zip(self): assert dagbag.get_dag("test_zip_dag")

[GitHub] [airflow] jedcunningham commented on a change in pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #21501: URL: https://github.com/apache/airflow/pull/21501#discussion_r804344322 ## File path: airflow/settings.py ## @@ -209,6 +209,18 @@ def get_airflow_context_vars(context): return {} +def

[GitHub] [airflow] github-actions[bot] commented on pull request #21503: update tutorial_etl_dag notes

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21503: URL: https://github.com/apache/airflow/pull/21503#issuecomment-1035863939 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] jedcunningham commented on a change in pull request #21503: update tutorial_etl_dag notes

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #21503: URL: https://github.com/apache/airflow/pull/21503#discussion_r804343873 ## File path: airflow/example_dags/tutorial_etl_dag.py ## @@ -19,9 +19,7 @@ """ ### ETL DAG Tutorial Documentation -This ETL DAG is

[GitHub] [airflow] leiguorui opened a new pull request #21510: Added haodf to the list of companies using Apache Airflow

2022-02-10 Thread GitBox
leiguorui opened a new pull request #21510: URL: https://github.com/apache/airflow/pull/21510 Added haodf to the list of companies using Airflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21510: Added haodf to the list of companies using Apache Airflow

2022-02-10 Thread GitBox
boring-cyborg[bot] commented on pull request #21510: URL: https://github.com/apache/airflow/pull/21510#issuecomment-1035861031 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] wolfier commented on issue #21395: Airflow Not Automatically Sending Errors to Sentry

2022-02-10 Thread GitBox
wolfier commented on issue #21395: URL: https://github.com/apache/airflow/issues/21395#issuecomment-1035857278 Which of `sentry-sdk` version is installed in the environment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] arkadiusz-bach edited a comment on issue #21225: Tasks stuck in queued state

2022-02-10 Thread GitBox
arkadiusz-bach edited a comment on issue #21225: URL: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930 I had similar problem, it was happening, because KubernetesExecutor is picking CeleryExecutor tasks on CeleryKubernetesExecutor Celery is changing task state

[GitHub] [airflow] chenglongyan commented on a change in pull request #21460: Add non-ascii character support in xcom table

2022-02-10 Thread GitBox
chenglongyan commented on a change in pull request #21460: URL: https://github.com/apache/airflow/pull/21460#discussion_r804323515 ## File path: tests/models/test_xcom.py ## @@ -377,6 +377,30 @@ def test_xcom_get_many_from_prior_dates_with_execution_date( assert

[GitHub] [airflow] ferruzzi commented on pull request #21422: [doc] Improve s3 operator example by adding task upload_keys

2022-02-10 Thread GitBox
ferruzzi commented on pull request #21422: URL: https://github.com/apache/airflow/pull/21422#issuecomment-1035790774 @zhongjiajie I say go ahead, it's non-breaking either way and I think we're going to focus on the ones that don't have any documentation or sample DAGs first so I don't see

[GitHub] [airflow] zhongjiajie commented on pull request #21422: [doc] Improve s3 operator example by adding task upload_keys

2022-02-10 Thread GitBox
zhongjiajie commented on pull request #21422: URL: https://github.com/apache/airflow/pull/21422#issuecomment-1035778852 @ferruzzi So should we merge or should we just close this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] zhongjiajie commented on pull request #21416: [doc] Fix copy path in S3 to redshift

2022-02-10 Thread GitBox
zhongjiajie commented on pull request #21416: URL: https://github.com/apache/airflow/pull/21416#issuecomment-1035754448 ping @mik-laj @potiuk @kaxil again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] github-actions[bot] commented on pull request #21507: Support config worker_enable_remote_control for celery

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21507: URL: https://github.com/apache/airflow/pull/21507#issuecomment-1035707166 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] github-actions[bot] commented on pull request #21509: Support globally override task run_as_user

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21509: URL: https://github.com/apache/airflow/pull/21509#issuecomment-1035706184 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] pingzh opened a new pull request #21509: Support globally override task run_as_user

2022-02-10 Thread GitBox
pingzh opened a new pull request #21509: URL: https://github.com/apache/airflow/pull/21509 It is very useful for the infra to set up their staging/dev airflow clusters and strictly control the task `run_as_user`. --- **^ Add meaningful description above** Read the

[GitHub] [airflow] ferruzzi commented on pull request #21508: Update EKS sample DAGs and docs

2022-02-10 Thread GitBox
ferruzzi commented on pull request #21508: URL: https://github.com/apache/airflow/pull/21508#issuecomment-1035685355 Fixing two of the three errors (my oops) but not sure about this one. ``` File

[GitHub] [airflow] ferruzzi edited a comment on pull request #21508: Update EKS sample DAGs and docs

2022-02-10 Thread GitBox
ferruzzi edited a comment on pull request #21508: URL: https://github.com/apache/airflow/pull/21508#issuecomment-1035685355 Fixing two of the three errors (my oops) but not sure about this one. ``` File

[GitHub] [airflow] derkuci commented on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
derkuci commented on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035680468 #20474 has been running in our system without issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] ferruzzi opened a new pull request #21508: Update EKS sample DAGs and docs

2022-02-10 Thread GitBox
ferruzzi opened a new pull request #21508: URL: https://github.com/apache/airflow/pull/21508 Part of a project to standardize AWS provider package sample DAGs and documentation. - Adds entries in the docs for the three EKS State Sensors - All EKS sample DAGs are reformatted to be

[GitHub] [airflow] pingzh commented on a change in pull request #21504: Set larger limit get_partitions_by_filter in HiveMetastoreHook

2022-02-10 Thread GitBox
pingzh commented on a change in pull request #21504: URL: https://github.com/apache/airflow/pull/21504#discussion_r804257340 ## File path: airflow/providers/apache/hive/hooks/hive.py ## @@ -556,7 +556,9 @@ def check_for_partition(self, schema: str, table: str, partition: str)

[GitHub] [airflow] pingzh opened a new pull request #21507: Support config worker_enable_remote_control for celery

2022-02-10 Thread GitBox
pingzh opened a new pull request #21507: URL: https://github.com/apache/airflow/pull/21507 so that we have control over it since when the value is `true`, it keeps creating `.*reply-celery-pidbox` queues if the flag is true

[GitHub] [airflow] arkadiusz-bach edited a comment on issue #21225: Tasks stuck in queued state

2022-02-10 Thread GitBox
arkadiusz-bach edited a comment on issue #21225: URL: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930 I had similar problem, it was happening, because KubernetesExecutor is picking CeleryExecutor tasks on CeleryKubernetesExecutor Celery is changing task state

[GitHub] [airflow] arkadiusz-bach edited a comment on issue #21225: Tasks stuck in queued state

2022-02-10 Thread GitBox
arkadiusz-bach edited a comment on issue #21225: URL: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930 I had similar problem, it was happening, because KubernetesExecutor is picking CeleryExecutor tasks on CeleryKubernetesExecutor Celery is changing task state

[GitHub] [airflow] arkadiusz-bach commented on issue #21225: Tasks stuck in queued state

2022-02-10 Thread GitBox
arkadiusz-bach commented on issue #21225: URL: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930 I had similar problem, it was happening, because KubernetesExecutor is picking CeleryExecutor tasks on CeleryKubernetesExecutor Celery is changing task state to

[GitHub] [airflow] eladkal commented on a change in pull request #21504: Set larger limit get_partitions_by_filter in HiveMetastoreHook

2022-02-10 Thread GitBox
eladkal commented on a change in pull request #21504: URL: https://github.com/apache/airflow/pull/21504#discussion_r804247073 ## File path: airflow/providers/apache/hive/hooks/hive.py ## @@ -556,7 +556,9 @@ def check_for_partition(self, schema: str, table: str, partition:

[GitHub] [airflow] kazanzhy commented on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
kazanzhy commented on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035649147 Changes in **Qubole** provider was in documentation (https://github.com/apache/airflow/pull/20058) and typing (https://github.com/apache/airflow/pull/21074). There are no

[GitHub] [airflow] jedcunningham commented on pull request #21462: Refactor airflow db init

2022-02-10 Thread GitBox
jedcunningham commented on pull request #21462: URL: https://github.com/apache/airflow/pull/21462#issuecomment-1035627808 Just for some context, this took `db init` on my machine from ~15 seconds to ~13 seconds. I was really hoping it'd make a bigger difference. My unscientific "I watched

[GitHub] [airflow] jedcunningham commented on a change in pull request #21478: Alt: Add a session backend to store session data in the database

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #21478: URL: https://github.com/apache/airflow/pull/21478#discussion_r804214993 ## File path: airflow/migrations/versions/c381b21cb7e4_add_session_table_to_db.py ## @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software

[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

2022-02-10 Thread GitBox
pierrejeambrun commented on pull request #20386: URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035619367 @potiuk Yes of course, I just rebased the branch. Nice I didn't know we could do that directly from the GitHub interface :+1: -- This is an automated message from

[GitHub] [airflow] turbaszek commented on a change in pull request #21267: Dataproc metastore assets

2022-02-10 Thread GitBox
turbaszek commented on a change in pull request #21267: URL: https://github.com/apache/airflow/pull/21267#discussion_r804208321 ## File path: airflow/providers/google/cloud/operators/dataproc_metastore.py ## @@ -29,13 +30,216 @@ from googleapiclient.errors import HttpError

[GitHub] [airflow] pingzh opened a new pull request #21506: Add celery_logging_level

2022-02-10 Thread GitBox
pingzh opened a new pull request #21506: URL: https://github.com/apache/airflow/pull/21506 so that we don't need to change the log level for tasks --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] turbaszek commented on pull request #21251: Google Cloud Composer opearators - with the deferrable option

2022-02-10 Thread GitBox
turbaszek commented on pull request #21251: URL: https://github.com/apache/airflow/pull/21251#issuecomment-1035615453 @lwyszomi please rebase to solve conflicts  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] potiuk edited a comment on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
potiuk edited a comment on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035596077 Cool. Loooking forward for more comments. We have time till ~ end of weekend (I will release on Sunday/Monday morning if I get enough votes) -- This is an automated

[GitHub] [airflow] potiuk commented on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
potiuk commented on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035596077 Cool. Loooking forward for more comments. We have time till ~ weekend (I will release on Sunday/Monday morning if I get enough votes) -- This is an automated message from the

[GitHub] [airflow] pierrejeambrun commented on issue #21443: Status of testing Providers that were prepared on February 09, 2022

2022-02-10 Thread GitBox
pierrejeambrun commented on issue #21443: URL: https://github.com/apache/airflow/issues/21443#issuecomment-1035589435 Hi, I tested all the methods for the new ZendeskHook, everything is working fine. #21349 :heavy_check_mark: -- This is an automated message from the Apache

[GitHub] [airflow] uranusjr opened a new pull request #21505: MappedOperator redesign

2022-02-10 Thread GitBox
uranusjr opened a new pull request #21505: URL: https://github.com/apache/airflow/pull/21505 This PR may be too large. Details to come later. Submitted for CI at the moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] github-actions[bot] commented on pull request #21504: Set larger limit get_partitions_by_filter in HiveMetastoreHook

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21504: URL: https://github.com/apache/airflow/pull/21504#issuecomment-1035571400 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] pingzh opened a new pull request #21504: Set larger limit get_partitions_by_filter in HiveMetastoreHook

2022-02-10 Thread GitBox
pingzh opened a new pull request #21504: URL: https://github.com/apache/airflow/pull/21504 With the large size of the hive metastore, setting limit to 1 will create a scan query that is too heavy to handle. Setting the limit to HiveMetastoreHook.MAX_PART_COUNT will generate a

[GitHub] [airflow] eladkal opened a new pull request #21503: update tutorial_etl_dag notes

2022-02-10 Thread GitBox
eladkal opened a new pull request #21503: URL: https://github.com/apache/airflow/pull/21503 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] josh-fell commented on pull request #21458: Bug Fix - S3DeleteObjectsOperator will try and delete all keys

2022-02-10 Thread GitBox
josh-fell commented on pull request #21458: URL: https://github.com/apache/airflow/pull/21458#issuecomment-1035547196 >This is because I assume templating does something with init so the check for this condition is not run. @njrs92 For some context here, the values of `XComArgs` or

[GitHub] [airflow] charmon79 commented on issue #15354: Error on DAG deletion : AttributeError: type object 'Base' has no attribute '_decl_class_registry'

2022-02-10 Thread GitBox
charmon79 commented on issue #15354: URL: https://github.com/apache/airflow/issues/15354#issuecomment-1035543726 I got this error just now when attempting to delete a DAG via the web UI, using Airflow 2.2.3 from the official Docker image (apache/airflow:2.2.3). -- This is an automated

[GitHub] [airflow] potiuk commented on issue #8194: Guide for Apache Pinot operators

2022-02-10 Thread GitBox
potiuk commented on issue #8194: URL: https://github.com/apache/airflow/issues/8194#issuecomment-1035528512 > Hi @eladkal , the example dags done on that pull request seem to have some errors, the arguments doesn't match that of the classes created in pinot.py Hmmm. Maybe fixing it

[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

2022-02-10 Thread GitBox
potiuk commented on pull request #20386: URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035527469 @mik-laj - any more comments on that ? @pierrejeambrun - can you please rebase to latest main to check if there are no static checks/doc changes (the change is 179 commits

[airflow] branch main updated (0d52e77 -> 5a6a2d6)

2022-02-10 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 0d52e77 Update docs/readme.rst (#21346) add 5a6a2d6 add how-to guide for pig operator (#21498) No new

[GitHub] [airflow] Melodie97 commented on issue #8194: Guide for Apache Pinot operators

2022-02-10 Thread GitBox
Melodie97 commented on issue #8194: URL: https://github.com/apache/airflow/issues/8194#issuecomment-1035523919 > > Add Guide for Apache Pinot #18930 > > Yes but still we can explain how to use with with PythonOperators using example dags. Some of the work was already done on #18930

[airflow] branch main updated (e052fb5 -> 0d52e77)

2022-02-10 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from e052fb5 Update Zendesk example DAG to use TaskFlow API (#21411) add 0d52e77 Update docs/readme.rst (#21346)

[GitHub] [airflow] potiuk merged pull request #21498: add how-to guide for pig operator

2022-02-10 Thread GitBox
potiuk merged pull request #21498: URL: https://github.com/apache/airflow/pull/21498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #8193: Guide for Apache Pig operators

2022-02-10 Thread GitBox
potiuk closed issue #8193: URL: https://github.com/apache/airflow/issues/8193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk merged pull request #21346: Update docs/readme.rst

2022-02-10 Thread GitBox
potiuk merged pull request #21346: URL: https://github.com/apache/airflow/pull/21346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on issue #8193: Guide for Apache Pig operators

2022-02-10 Thread GitBox
potiuk commented on issue #8193: URL: https://github.com/apache/airflow/issues/8193#issuecomment-1035521571 Assigned you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] potiuk commented on a change in pull request #20962: Support generating SQL script for upgrades

2022-02-10 Thread GitBox
potiuk commented on a change in pull request #20962: URL: https://github.com/apache/airflow/pull/20962#discussion_r804108700 ## File path: airflow/utils/db.py ## @@ -1022,8 +1026,24 @@ def _validate_version_range(command, config, version_range): 'Please provide

[GitHub] [airflow] potiuk commented on a change in pull request #20962: Support generating SQL script for upgrades

2022-02-10 Thread GitBox
potiuk commented on a change in pull request #20962: URL: https://github.com/apache/airflow/pull/20962#discussion_r804108700 ## File path: airflow/utils/db.py ## @@ -1022,8 +1026,24 @@ def _validate_version_range(command, config, version_range): 'Please provide

[GitHub] [airflow] mluscon commented on issue #21487: Requiring sudo privilages

2022-02-10 Thread GitBox
mluscon commented on issue #21487: URL: https://github.com/apache/airflow/issues/21487#issuecomment-1035480216 > Looked a bit closer - I assume you only want to add the "airflow" user a capability to only sudo to "run_as_user" but not to root? Seems doable. Exactly, my "airflow"

[airflow] branch main updated (51aff27 -> e052fb5)

2022-02-10 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 51aff27 Optionally raise an error if source file does not exist in GCSToGCSOperator (#21391) add e052fb5

[GitHub] [airflow] eladkal merged pull request #21411: Update Zendesk example DAG to use TaskFlow API

2022-02-10 Thread GitBox
eladkal merged pull request #21411: URL: https://github.com/apache/airflow/pull/21411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #21411: Update Zendesk example DAG to use TaskFlow API

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21411: URL: https://github.com/apache/airflow/pull/21411#issuecomment-1035464765 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

2022-02-10 Thread GitBox
pierrejeambrun commented on pull request #20386: URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035458565 Hi @eladkal and @potiuk, It has been a couple of week since last activity on this PR. I would like to follow up, I think it's ready to merge. Is there

[GitHub] [airflow] Melodie97 commented on pull request #21498: add how-to guide for pig operator

2022-02-10 Thread GitBox
Melodie97 commented on pull request #21498: URL: https://github.com/apache/airflow/pull/21498#issuecomment-1035412045 @potiuk , pls review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] Melodie97 commented on issue #8193: Guide for Apache Pig operators

2022-02-10 Thread GitBox
Melodie97 commented on issue #8193: URL: https://github.com/apache/airflow/issues/8193#issuecomment-1035406316 I'll like to work on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] github-actions[bot] commented on pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21501: URL: https://github.com/apache/airflow/pull/21501#issuecomment-1035404183 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] github-actions[bot] commented on pull request #21445: [de]serialize resources on task correctly

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21445: URL: https://github.com/apache/airflow/pull/21445#issuecomment-1035403752 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] potiuk commented on issue #21087: KubernetesJobWatcher failing on HTTP 410 errors, jobs stuck in scheduled state

2022-02-10 Thread GitBox
potiuk commented on issue #21087: URL: https://github.com/apache/airflow/issues/21087#issuecomment-1035396420 > More context here: [#15500 (comment)](https://github.com/apache/airflow/pull/15500#issuecomment-827885109) > > Bottom line, especially now that we aren't pinned to

[GitHub] [airflow] pingzh opened a new pull request #21501: Support different timeout value for dag file parsing

2022-02-10 Thread GitBox
pingzh opened a new pull request #21501: URL: https://github.com/apache/airflow/pull/21501 This adds a hook for infra users to dynamically control the dag file parsing timeout. It is useful when there are a few dag files, requiring long parsing time while others not You

[airflow] branch main updated (a2abf66 -> 51aff27)

2022-02-10 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from a2abf66 Fix postgres hook import pipeline tutorial (#21491) add 51aff27 Optionally raise an error if source

[GitHub] [airflow] potiuk merged pull request #21391: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
potiuk merged pull request #21391: URL: https://github.com/apache/airflow/pull/21391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #21388: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
potiuk closed issue #21388: URL: https://github.com/apache/airflow/issues/21388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21391: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
boring-cyborg[bot] commented on pull request #21391: URL: https://github.com/apache/airflow/pull/21391#issuecomment-1035392189 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk commented on pull request #21391: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
potiuk commented on pull request #21391: URL: https://github.com/apache/airflow/pull/21391#issuecomment-1035391963 It worked indeed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ferruzzi edited a comment on pull request #21423: Refactor query status polling logic in EMRContainerHook

2022-02-10 Thread GitBox
ferruzzi edited a comment on pull request #21423: URL: https://github.com/apache/airflow/pull/21423#issuecomment-1035352431 I like it. You'll have to wait for a Committer to approve and merge, but I don't see any issues. Maybe edit the top post to link to the Athena one as well.

[GitHub] [airflow] jedcunningham commented on a change in pull request #20962: Support generating SQL script for upgrades

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #20962: URL: https://github.com/apache/airflow/pull/20962#discussion_r804025901 ## File path: airflow/utils/db.py ## @@ -1022,8 +1026,24 @@ def _validate_version_range(command, config, version_range): 'Please

[GitHub] [airflow] jedcunningham commented on a change in pull request #20962: Support generating SQL script for upgrades

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #20962: URL: https://github.com/apache/airflow/pull/20962#discussion_r804025901 ## File path: airflow/utils/db.py ## @@ -1022,8 +1026,24 @@ def _validate_version_range(command, config, version_range): 'Please

[GitHub] [airflow] dstandish commented on pull request #21500: S3KeySensor should parse bucket and key outside of poke

2022-02-10 Thread GitBox
dstandish commented on pull request #21500: URL: https://github.com/apache/airflow/pull/21500#issuecomment-1035362954 > Parsing `bucket_key` in the `__init__()` won't work if the value is an `XComArg` or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of

[GitHub] [airflow] josh-fell edited a comment on pull request #21500: S3KeySensor should parse bucket and key outside of poke

2022-02-10 Thread GitBox
josh-fell edited a comment on pull request #21500: URL: https://github.com/apache/airflow/pull/21500#issuecomment-1035357422 FWIW, datatype checks on templated fields within `__init__()` have their issues as well. See #20347. -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] josh-fell commented on pull request #21500: S3KeySensor should parse bucket and key outside of poke

2022-02-10 Thread GitBox
josh-fell commented on pull request #21500: URL: https://github.com/apache/airflow/pull/21500#issuecomment-1035357422 FWIW, datatype checks on templated fields have their issues within `__init__()` as well. See #20347. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] ferruzzi commented on pull request #21423: Refactor query status polling logic in EMRContainerHook

2022-02-10 Thread GitBox
ferruzzi commented on pull request #21423: URL: https://github.com/apache/airflow/pull/21423#issuecomment-1035352431 I like it. You'll have to wait for a Committer to approve and merge, but I don't see any issues. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] jedcunningham commented on a change in pull request #21222: Helm: Fix elasticsearch URL in secret when username/password fields are empty,

2022-02-10 Thread GitBox
jedcunningham commented on a change in pull request #21222: URL: https://github.com/apache/airflow/pull/21222#discussion_r804016587 ## File path: chart/tests/test_elasticsearch_secret.py ## @@ -132,3 +132,30 @@ def test_should_generate_secret_with_specified_schemes(self,

[GitHub] [airflow] github-actions[bot] commented on pull request #21222: Helm: Fix elasticsearch URL in secret when username/password fields are empty,

2022-02-10 Thread GitBox
github-actions[bot] commented on pull request #21222: URL: https://github.com/apache/airflow/pull/21222#issuecomment-1035348537 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] josh-fell commented on pull request #21500: S3KeySensor should parse bucket and key outside of poke

2022-02-10 Thread GitBox
josh-fell commented on pull request #21500: URL: https://github.com/apache/airflow/pull/21500#issuecomment-1035347722 Parsing `bucket_key` in the `__init__()` won't work if the value is an `XComArg` or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of

[GitHub] [airflow] potiuk closed pull request #21391: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
potiuk closed pull request #21391: URL: https://github.com/apache/airflow/pull/21391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #21391: Optionally raise an error if source file does not exist in GCSToGCSOperator

2022-02-10 Thread GitBox
potiuk commented on pull request #21391: URL: https://github.com/apache/airflow/pull/21391#issuecomment-1035338818 Not entirely it seems:). Let me close/reopen to rebuild. I think we need to add a trrigger for another even in our CI :) -- This is an automated message from the Apache Git

[GitHub] [airflow] jedcunningham commented on issue #21087: KubernetesJobWatcher failing on HTTP 410 errors, jobs stuck in scheduled state

2022-02-10 Thread GitBox
jedcunningham commented on issue #21087: URL: https://github.com/apache/airflow/issues/21087#issuecomment-1035327244 It's not that there are too many changes, at least like you are thinking. The executor is trying to watch from (history) revision n, which has rolled off of history on the

[airflow] 17/18: Fix mismatch in generated run_id and logical date of DAG run (#18707)

2022-02-10 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch v2-2-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 7aae75c140a6509b8840b6f0dc5a9d0c95f6d81a Author: David Caron AuthorDate: Thu Feb 3 21:14:19 2022 -0500

[airflow] 18/18: Fix postgres hook import pipeline tutorial (#21491)

2022-02-10 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch v2-2-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 7c93cf73420683f7974f3a65c034f7de28a816ea Author: KevinYanesG <75472729+kevinyan...@users.noreply.github.com>

  1   2   3   >