Re: [PR] NeptuneStopDbClusterOperator - Handle invalid cluster states [airflow]

2024-05-21 Thread via GitHub
eladkal commented on code in PR #38287: URL: https://github.com/apache/airflow/pull/38287#discussion_r1607748032 ## airflow/providers/amazon/aws/operators/neptune.py: ## @@ -32,6 +36,50 @@ from airflow.utils.context import Context +def handle_waitable_exception( +

[PR] remove deprecation drill [airflow]

2024-05-21 Thread via GitHub
Bowrna opened a new pull request, #39732: URL: https://github.com/apache/airflow/pull/39732 Related: #39485 Removing deprecated DrillOperator from docs and tests and replacing with SQLExecuteQueryOperator --- **^ Add meaningful description above**

Re: [I] Issues with configuring airflow 2.6.3/python3.11 with LDAP [airflow]

2024-05-21 Thread via GitHub
eladkal closed issue #39379: Issues with configuring airflow 2.6.3/python3.11 with LDAP URL: https://github.com/apache/airflow/issues/39379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] remove deprecation sqlite [airflow]

2024-05-21 Thread via GitHub
Bowrna opened a new pull request, #39730: URL: https://github.com/apache/airflow/pull/39730 Related: #39485 Removing deprecated SqliteOperator from docs and tests and replacing with SQLExecuteQueryOperator --- **^ Add meaningful description above**

[PR] remove deprecations snowflake [airflow]

2024-05-21 Thread via GitHub
Bowrna opened a new pull request, #39729: URL: https://github.com/apache/airflow/pull/39729 Related: https://github.com/apache/airflow/issues/39485 Removing deprecated SnowflakeOperator from docs and tests and replacing with SQLExecuteQueryOperator ---

Re: [I] Resolve `AirflowProviderDeprecationWarning` in providers system tests [airflow]

2024-05-21 Thread via GitHub
Bowrna commented on issue #39485: URL: https://github.com/apache/airflow/issues/39485#issuecomment-2121816874 Raising PR for Snowflake and SQLite too! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Resolving mysql deprecated operator warnings [airflow]

2024-05-21 Thread via GitHub
Bowrna commented on PR #39725: URL: https://github.com/apache/airflow/pull/39725#issuecomment-2121815950 @dirrao you may need to edit the documentation part too airflow/docs/apache-airflow-providers-mysql/operators.rst -- This is an automated message from the Apache Git Service. To

[PR] remove deprecation trino [airflow]

2024-05-21 Thread via GitHub
Bowrna opened a new pull request, #39728: URL: https://github.com/apache/airflow/pull/39728 Related: #39485 Removing deprecated TrinoOperator from docs and tests and replacing with SQLExecuteQueryOperator --- **^ Add meaningful description

Re: [I] Resolve `AirflowProviderDeprecationWarning` in providers system tests [airflow]

2024-05-21 Thread via GitHub
Bowrna commented on issue #39485: URL: https://github.com/apache/airflow/issues/39485#issuecomment-2121813081 > I am going to raise PR for Trino. oops @dirrao I am sorry I raised a PR without noticing it :( -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] removing deprecation postgres [airflow]

2024-05-20 Thread via GitHub
Bowrna commented on PR #39706: URL: https://github.com/apache/airflow/pull/39706#issuecomment-2121804101 @Taragolis I have added the warning part in docs. Let me know if it looks good now. Will do the same for the other DB related providers too! -- This is an automated message from the

Re: [PR] Avoid resetting adopted task instances when retrying for kubernetes executor [airflow]

2024-05-20 Thread via GitHub
tanvn commented on PR #39406: URL: https://github.com/apache/airflow/pull/39406#issuecomment-2121761730 @jedcunningham @hussein-awala ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] removing deprecations gke [airflow]

2024-05-20 Thread via GitHub
Bowrna opened a new pull request, #39727: URL: https://github.com/apache/airflow/pull/39727 Related: #39485 --- **^ Add meaningful description above** Read the **[Pull Request

[PR] gcs to gcs deprecation removal [airflow]

2024-05-20 Thread via GitHub
Bowrna opened a new pull request, #39726: URL: https://github.com/apache/airflow/pull/39726 Related: #39485 --- **^ Add meaningful description above** Read the **[Pull Request

[PR] Resolving mysql deprecated operator warnings [airflow]

2024-05-20 Thread via GitHub
dirrao opened a new pull request, #39725: URL: https://github.com/apache/airflow/pull/39725 Resolving mysql deprecated operator warnings related: #39485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Raise error upon DAG import instead of DAG trigger for incorrect pool_slots value in mapped tasks using partial() [airflow]

2024-05-20 Thread via GitHub
karenbraganz opened a new pull request, #39724: URL: https://github.com/apache/airflow/pull/39724 Check pool_slots value in mapped tasks using partial() --- Issue #39639 was opened because a DAG import error would show up for regular tasks with an incorrect

Re: [I] sla_miss_callback is not working with 5 input parameters as official document, only dag_id 1 input is allowed, no enough context for alert [airflow]

2024-05-20 Thread via GitHub
github-actions[bot] commented on issue #38931: URL: https://github.com/apache/airflow/issues/38931#issuecomment-2121467063 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] sla_miss_callback is not working with 5 input parameters as official document, only dag_id 1 input is allowed, no enough context for alert [airflow]

2024-05-20 Thread via GitHub
github-actions[bot] closed issue #38931: sla_miss_callback is not working with 5 input parameters as official document, only dag_id 1 input is allowed, no enough context for alert URL: https://github.com/apache/airflow/issues/38931 -- This is an automated message from the Apache Git

Re: [I] EcsRunTaskOperator does not send logs if task times out [airflow]

2024-05-20 Thread via GitHub
mateuslatrova commented on issue #39026: URL: https://github.com/apache/airflow/issues/39026#issuecomment-2121387562 > > Hi @slycyberguy , are you still handling this issue? I am also interested in it, in case you need any help. > > Hi @mateuslatrova. It looks like I'm getting

Re: [I] Dynamic task mapping - partial args are not check and only fail at trigger [airflow]

2024-05-20 Thread via GitHub
RNHTTR commented on issue #39639: URL: https://github.com/apache/airflow/issues/39639#issuecomment-2121383804 Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Render map_index_template when the task status is set manually [airflow]

2024-05-20 Thread via GitHub
karenbraganz closed pull request #39505: Render map_index_template when the task status is set manually URL: https://github.com/apache/airflow/pull/39505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] More typing in TimeSensor and TimeSensorAsync [airflow]

2024-05-20 Thread via GitHub
jedcunningham merged PR #39696: URL: https://github.com/apache/airflow/pull/39696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] More typing in TimeSensor and TimeSensorAsync [airflow]

2024-05-20 Thread via GitHub
jedcunningham commented on code in PR #39696: URL: https://github.com/apache/airflow/pull/39696#discussion_r1607344807 ## airflow/sensors/time_sensor.py: ## @@ -72,13 +72,13 @@ def __init__(self, *, target_time, **kwargs): self.target_datetime =

[PR] Load Provider Plugins in Development [airflow]

2024-05-20 Thread via GitHub
jscheffl opened a new pull request, #39723: URL: https://github.com/apache/airflow/pull/39723 As I am starting to build a PoC and experience code in and for AIP-69 I realized that provider packages from the mono-repo do not load plugins when starting the system via breeze. Plugins are

Re: [I] Dynamic task mapping - partial args are not check and only fail at trigger [airflow]

2024-05-20 Thread via GitHub
karenbraganz commented on issue #39639: URL: https://github.com/apache/airflow/issues/39639#issuecomment-2121258722 I would like to be assigned this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-20 Thread via GitHub
raphaelauv commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1607279904 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Determine needs_expansion at time of serialization [airflow]

2024-05-20 Thread via GitHub
dstandish merged PR #39604: URL: https://github.com/apache/airflow/pull/39604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-20 Thread via GitHub
dondaum commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1607267994 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Fixed issue of new dag getting old dataset events. [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39603: URL: https://github.com/apache/airflow/pull/39603#discussion_r1607251286 ## tests/jobs/test_scheduler_job.py: ## @@ -3869,13 +3870,118 @@ def dict_from_obj(obj): assert created_run.data_interval_start == DEFAULT_DATE +

Re: [PR] Fixed issue of new dag getting old dataset events. [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39603: URL: https://github.com/apache/airflow/pull/39603#discussion_r1607251286 ## tests/jobs/test_scheduler_job.py: ## @@ -3869,13 +3870,118 @@ def dict_from_obj(obj): assert created_run.data_interval_start == DEFAULT_DATE +

Re: [PR] Fixed issue of new dag getting old dataset events. [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39603: URL: https://github.com/apache/airflow/pull/39603#discussion_r1607250735 ## tests/jobs/test_scheduler_job.py: ## @@ -3869,13 +3870,118 @@ def dict_from_obj(obj): assert created_run.data_interval_start == DEFAULT_DATE +

Re: [PR] Fixed issue of new dag getting old dataset events. [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39603: URL: https://github.com/apache/airflow/pull/39603#discussion_r1607249732 ## tests/jobs/test_scheduler_job.py: ## @@ -3869,13 +3870,118 @@ def dict_from_obj(obj): assert created_run.data_interval_start == DEFAULT_DATE +

Re: [PR] Fixed issue of new dag getting old dataset events. [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39603: URL: https://github.com/apache/airflow/pull/39603#discussion_r1607248871 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1277,6 +1277,10 @@ def _create_dag_runs_dataset_triggered( ] if previous_dag_run:

[I] `application/octet-stream` content-type when writing to Azure Blob Storage using `ObjectStoragePath` [airflow]

2024-05-20 Thread via GitHub
pedro-cf opened a new issue, #39722: URL: https://github.com/apache/airflow/issues/39722 ### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When using `ObjectStoragePath` to send a file to

Re: [PR] Run tests for Providers also for Airflow 2.8 [airflow]

2024-05-20 Thread via GitHub
bolkedebruin commented on PR #39606: URL: https://github.com/apache/airflow/pull/39606#issuecomment-2121092524 @potiuk Airflow 2.8.4 requires (constraints) universal-path 0.1.4 which is indeed not compatible with python 3.12. However., I am not sure if this is related. The `namespace`

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-20 Thread via GitHub
kahlstrm commented on PR #39496: URL: https://github.com/apache/airflow/pull/39496#issuecomment-2121036687 @eladkal fixed the tests, seems like the odd behavior with `TaskInstanceState.DEFERRED` was fixed, and me working around it in these tests were broken due to that. -- This is an

Re: [PR] Fix Pod Number Exception in the sync mode if reattach_on_restart parameter is False [airflow]

2024-05-20 Thread via GitHub
shahar1 commented on code in PR #39329: URL: https://github.com/apache/airflow/pull/39329#discussion_r1607151894 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -533,24 +539,31 @@ def find_pod(self, namespace: str, context: Context, *, exclude_checked: bool =

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-20 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1607103303 ## airflow/providers/dbt/cloud/CHANGELOG.rst: ## @@ -28,6 +28,11 @@ Changelog - +main +. + +In Airflow 2.10.0, we fix the way try_number works, so

Re: [PR] Home: filter running/failed and active/paused dags [airflow]

2024-05-20 Thread via GitHub
RNHTTR commented on code in PR #39701: URL: https://github.com/apache/airflow/pull/39701#discussion_r1607086380 ## airflow/www/views.py: ## @@ -785,13 +787,25 @@ def index(self): # Remove the reset_tags=reset from the URL return

Re: [PR] [openlineage] Update conf retrieval docstring and adjust pool_size [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39721: URL: https://github.com/apache/airflow/pull/39721#discussion_r1607079331 ## airflow/providers/openlineage/conf.py: ## @@ -14,6 +14,19 @@ # KIND, either express or implied. See the License for the # specific language governing

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-05-20 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1607061565 ## airflow/models/taskinstance.py: ## @@ -1003,25 +1009,40 @@ def _refresh_from_task( task_instance_mutation_hook(task_instance) +@internal_api_call

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-05-20 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1607059883 ## airflow/models/taskinstance.py: ## @@ -1003,25 +1009,40 @@ def _refresh_from_task( task_instance_mutation_hook(task_instance) +@internal_api_call

Re: [PR] Home: filter running/failed and active/paused dags [airflow]

2024-05-20 Thread via GitHub
ecodina commented on PR #39701: URL: https://github.com/apache/airflow/pull/39701#issuecomment-2120878422 @bbovenzi done! Similarly to the tags I added the "reset_filter" param to reset the cookie when we deselect the filter. However, the user does not see it, since it immediately

Re: [PR] Power BI Provider [airflow]

2024-05-20 Thread via GitHub
ambika-garg commented on PR #39243: URL: https://github.com/apache/airflow/pull/39243#issuecomment-2120876679 Agreed, I'll submit this PR for the Power BI operator under microsoft.azure (initially, I mistakenly mentioned Azure Data Factory). So, let's go ahead and close this PR for now.

Re: [PR] [openlineage] Update conf retrieval docstring and adjust pool_size [airflow]

2024-05-20 Thread via GitHub
kacpermuda commented on PR #39721: URL: https://github.com/apache/airflow/pull/39721#issuecomment-2120845371 @JDarDagran ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [openlineage] Update conf retrieval docstring and adjust pool_size [airflow]

2024-05-20 Thread via GitHub
kacpermuda opened a new pull request, #39721: URL: https://github.com/apache/airflow/pull/39721 Adding a docstring to conf.py describing how we should retrieve OL configuration. Adjusting `dag_state_change_process_pool_size` to match that logic. --- **^ Add

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1607024896 ## airflow/models/taskinstance.py: ## @@ -1003,25 +1009,40 @@ def _refresh_from_task( task_instance_mutation_hook(task_instance) +@internal_api_call

Re: [PR] Determine needs_expansion at time of serialization [airflow]

2024-05-20 Thread via GitHub
dstandish commented on code in PR #39604: URL: https://github.com/apache/airflow/pull/39604#discussion_r1607023336 ## airflow/serialization/serialized_objects.py: ## @@ -553,6 +553,7 @@ def serialize( elif isinstance(var, MappedOperator): return

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-20 Thread via GitHub
kahlstrm commented on PR #39496: URL: https://github.com/apache/airflow/pull/39496#issuecomment-2120824638 > @kahlstrm can you fix the failing tests? Sure, didn't realize the tests failing are related ones, one sec. -- This is an automated message from the Apache Git Service. To

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
dstandish commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120823774 also for reference @jscheffl here's the PRs that came before on this one  https://github.com/apache/airflow/pulls?q=is%3Apr+project%3Aapache%2F169+is%3Aclosed -- This is

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
dstandish commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120822126 > So as you cherry-pick into smaller PRs and leaving complexity manageable... if all is green might still be good to merge the pieces and I assume to make it working then a few other

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
dstandish commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120816036 > missing in internal API or will Subdag just not be supported? + etc ah yeah those scenarios are good to add to my testing dag. maybe you could share your dag? and maybe i

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
jscheffl commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120812783 > thanks @jscheffl -- which branch were you on? there are more changes re mapping on the "mothership" PR (#37851) that are excluded from this PR just for easier review -- that's the

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
dstandish commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120782203 thanks @jscheffl -- which branch were you on? there are more changes re mapping on the "mothership" PR (https://github.com/apache/airflow/pull/37851) that are excluded from this PR

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-20 Thread via GitHub
jscheffl commented on PR #38992: URL: https://github.com/apache/airflow/pull/38992#issuecomment-2120775487 Okay, did some testing with a breeze setup, manually "massaged" the ENV to enable AIP-44, strated internal API server, took away the DB connection and set the internal API endpoints

Re: [I] Resolve `AirflowProviderDeprecationWarning` in providers system tests [airflow]

2024-05-20 Thread via GitHub
dirrao commented on issue #39485: URL: https://github.com/apache/airflow/issues/39485#issuecomment-2120770948 I am going to raise PR for Trino. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Home: filter running/failed and active/paused dags [airflow]

2024-05-20 Thread via GitHub
bbovenzi commented on PR #39701: URL: https://github.com/apache/airflow/pull/39701#issuecomment-2120769740 > > Might it be cleaner to just unselect failed? There are only two buttons, so I don't know that it makes sense to include an "all states" button when there are only two states to

Re: [PR] Determine needs_expansion at time of serialization [airflow]

2024-05-20 Thread via GitHub
dstandish commented on code in PR #39604: URL: https://github.com/apache/airflow/pull/39604#discussion_r1606973017 ## airflow/serialization/serialized_objects.py: ## @@ -553,6 +553,7 @@ def serialize( elif isinstance(var, MappedOperator): return

Re: [PR] openlineage: remove `openlineage.common` dependencies in Google and Snowflake providers. [airflow]

2024-05-20 Thread via GitHub
mobuchowski merged PR #39614: URL: https://github.com/apache/airflow/pull/39614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Reraise exception from strict dataset URI checks [airflow]

2024-05-20 Thread via GitHub
uranusjr merged PR #39719: URL: https://github.com/apache/airflow/pull/39719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] More typing in TimeSensor and TimeSensorAsync [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39696: URL: https://github.com/apache/airflow/pull/39696#discussion_r1606959336 ## airflow/sensors/time_sensor.py: ## @@ -72,13 +72,13 @@ def __init__(self, *, target_time, **kwargs): self.target_datetime =

Re: [PR] UI changes for DAG Reparsing feature [airflow]

2024-05-20 Thread via GitHub
bbovenzi commented on PR #39636: URL: https://github.com/apache/airflow/pull/39636#issuecomment-2120729661 > @utkarsharma2 yes you are right > > then this ? > >

Re: [I] Audit task failed-deps for later investigation [airflow]

2024-05-20 Thread via GitHub
eladkal commented on issue #39712: URL: https://github.com/apache/airflow/issues/39712#issuecomment-2120696473 > Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-20 Thread via GitHub
eladkal commented on PR #39496: URL: https://github.com/apache/airflow/pull/39496#issuecomment-2120655139 @kahlstrm can you fix the failing tests? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] KPO - get_logs - logs missing [airflow]

2024-05-20 Thread via GitHub
Lee2532 commented on issue #39682: URL: https://github.com/apache/airflow/issues/39682#issuecomment-2120627003 What happen if you change your script to? print to logging import time import logging if __name__ == '__main__': for i in range(10): time.sleep(2)

[PR] Reraise exception from strict dataset URI checks [airflow]

2024-05-20 Thread via GitHub
uranusjr opened a new pull request, #39719: URL: https://github.com/apache/airflow/pull/39719 Minor improvement from #39670. Instead of `raise exception`, re-raise the original exception instead to preserve the error stack. Code under the raise is unindented instead of using `else`. --

Re: [PR] Determine needs_expansion at time of serialization [airflow]

2024-05-20 Thread via GitHub
uranusjr commented on code in PR #39604: URL: https://github.com/apache/airflow/pull/39604#discussion_r1606822943 ## airflow/serialization/serialized_objects.py: ## @@ -553,6 +553,7 @@ def serialize( elif isinstance(var, MappedOperator): return

Re: [PR] Refactor Datapipeline operators and fix system tests [airflow]

2024-05-20 Thread via GitHub
VladaZakharova closed pull request #39716: Refactor Datapipeline operators and fix system tests URL: https://github.com/apache/airflow/pull/39716 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Deployment: Airflow 2.9.1 Installed on Kubernetes via Helm Chart - Opensearch Connection Issue (Error 406) [airflow]

2024-05-20 Thread via GitHub
adrienyhuel commented on issue #39718: URL: https://github.com/apache/airflow/issues/39718#issuecomment-2120444203 Hello @emirhanaydin61 It is related to #33619 Airflow use "apache-airflow-providers-elasticsearch" to read logs, and the latests versions aren't compatible with

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-20 Thread via GitHub
jonathan-ostrander commented on code in PR #39694: URL: https://github.com/apache/airflow/pull/39694#discussion_r1606765260 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -842,16 +838,14 @@ def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod): if

Re: [I] Deployment: Airflow 2.9.1 Installed on Kubernetes via Helm Chart - Opensearch Connection Issue (Error 406) [airflow]

2024-05-20 Thread via GitHub
boring-cyborg[bot] commented on issue #39718: URL: https://github.com/apache/airflow/issues/39718#issuecomment-2120412681 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Deployment: Airflow 2.9.1 Installed on Kubernetes via Helm Chart - Opensearch Connection Issue (Error 406) [airflow]

2024-05-20 Thread via GitHub
emirhanaydin61 opened a new issue, #39718: URL: https://github.com/apache/airflow/issues/39718 ### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Hello, we are using airflow 2.4.0 and were

Re: [PR] Refactor Datapipeline operators and fix system tests [airflow]

2024-05-20 Thread via GitHub
VladaZakharova commented on code in PR #39716: URL: https://github.com/apache/airflow/pull/39716#discussion_r1606732810 ## tests/deprecations_ignore.yml: ## @@ -395,6 +395,9 @@ -

Re: [PR] Run tests for Providers also for Airflow 2.8 [airflow]

2024-05-20 Thread via GitHub
potiuk commented on PR #39606: URL: https://github.com/apache/airflow/pull/39606#issuecomment-2120356096 > together with OL support, so I'm trying to understand if there might be some mismatch of the provider versions there. I think the problem is connected with upath integration.

Re: [PR] Run tests for Providers also for Airflow 2.8 [airflow]

2024-05-20 Thread via GitHub
potiuk commented on PR #39606: URL: https://github.com/apache/airflow/pull/39606#issuecomment-2120352219 > @potiuk Could you please tell me which command should i run to reproduce? Are we running the providers from current main against different airflow versions or are we using some past

[I] Executor reports task instance (...) finished (failed) although the task says it's queued [airflow]

2024-05-20 Thread via GitHub
andreyvital opened a new issue, #39717: URL: https://github.com/apache/airflow/issues/39717 ### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? ``` [2024-05-20T12:03:24.184+]

Re: [I] Audit task failed-deps for later investigation [airflow]

2024-05-20 Thread via GitHub
tirkarthi commented on issue #39712: URL: https://github.com/apache/airflow/issues/39712#issuecomment-2120336770 Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched

Re: [PR] Add watchdog for immediately processing changes in the DAGs folder [airflow]

2024-05-20 Thread via GitHub
BasPH commented on PR #34487: URL: https://github.com/apache/airflow/pull/34487#issuecomment-2120314813 Hi @jscheffl, unfortunately life got in the way of investing time in OSS work the last year. The essentials are there but I was working on integrating this code nicely with

Re: [I] Unable to see dag_id and task_id tagging for mentioned metrics [airflow]

2024-05-20 Thread via GitHub
EMT-Swaroop commented on issue #39522: URL: https://github.com/apache/airflow/issues/39522#issuecomment-2120313306 @dirrao , we didn't customize any configuration in the statsd deployment, we are using the stasd exporter which coming from the parent airflow chart (1.13.1) -- This is an

Re: [PR] UI changes for DAG Reparsing feature [airflow]

2024-05-20 Thread via GitHub
raphaelauv commented on PR #39636: URL: https://github.com/apache/airflow/pull/39636#issuecomment-2120304664 @utkarsharma2 yes you are right then this ? ![image](https://github.com/apache/airflow/assets/10202690/01c64c56-3ec4-4760-9819-67ba6fe97c8f) -- This is an

Re: [PR] Refactor Datapipeline operators and fix system tests [airflow]

2024-05-20 Thread via GitHub
eladkal commented on code in PR #39716: URL: https://github.com/apache/airflow/pull/39716#discussion_r1606679948 ## tests/deprecations_ignore.yml: ## @@ -395,6 +395,9 @@ -

[PR] Refactor Datapipeline operators and fix system tests [airflow]

2024-05-20 Thread via GitHub
VladaZakharova opened a new pull request, #39716: URL: https://github.com/apache/airflow/pull/39716 This PR: Removes datapipeline module from Google provider Moves all the related operators inside Dataflow service, based on the API that is used inside operators Adds new

Re: [I] SQLSensor does not time out [airflow]

2024-05-20 Thread via GitHub
eladkal commented on issue #39219: URL: https://github.com/apache/airflow/issues/39219#issuecomment-2120292442 > Some of the SQL Sensors from time to time does not respect the timeout interval and run the entire DAG for multiple days. The sensors we use have waiting timeout 12 hours and

Re: [I] SQLSensor does not time out [airflow]

2024-05-20 Thread via GitHub
eladkal commented on issue #39219: URL: https://github.com/apache/airflow/issues/39219#issuecomment-2120286663 > Unfortunately it looks like it is a locked down parameter :-( > > Amazon MWAA reserves a set of critical environment variables. If you overwrite a reserved variable,

Re: [I] webserver - acces task log | TypeError: '<' not supported between instances of 'Interval' and 'DateTime' [airflow]

2024-05-20 Thread via GitHub
eladkal commented on issue #39306: URL: https://github.com/apache/airflow/issues/39306#issuecomment-2120283215 Not sure what we can do here without reproduce example -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] data to long when using s3 to gcs [airflow]

2024-05-20 Thread via GitHub
eladkal closed issue #39708: data to long when using s3 to gcs URL: https://github.com/apache/airflow/issues/39708 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Bump version of atlassian-python-api [airflow]

2024-05-20 Thread via GitHub
eladkal opened a new pull request, #39714: URL: https://github.com/apache/airflow/pull/39714 Bump version to avoid workaround of taking direct dependency on `beautifulsoup4` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] UI changes for DAG Reparsing feature [airflow]

2024-05-20 Thread via GitHub
utkarsharma2 commented on PR #39636: URL: https://github.com/apache/airflow/pull/39636#issuecomment-2120227411 > maybe something like > >

Re: [PR] Fix acknowledged functionality in deferrable mode for PubSubPullSensor [airflow]

2024-05-20 Thread via GitHub
MaksYermak commented on PR #39711: URL: https://github.com/apache/airflow/pull/39711#issuecomment-2120222492 > > It is needed to fix the issue in acknowledged functionality in deferrable mode. > > If this is a bug fix then we should have a unit test to avoid regression

Re: [PR] remove deprecations google cloud [airflow]

2024-05-20 Thread via GitHub
Bowrna commented on PR #39713: URL: https://github.com/apache/airflow/pull/39713#issuecomment-2120193949 I will add the PR for other google cloud examples as each of the fix in it have different set of doubts. Raising multiple PR to avoid confusion -- This is an automated message from

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-20 Thread via GitHub
eladkal commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1606589165 ## airflow/providers/dbt/cloud/CHANGELOG.rst: ## @@ -28,6 +28,11 @@ Changelog - +main +. + +In Airflow 2.10.0, we fix the way try_number works, so

[PR] remove deprecations google cloud [airflow]

2024-05-20 Thread via GitHub
Bowrna opened a new pull request, #39713: URL: https://github.com/apache/airflow/pull/39713 Related: #39485 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Fix acknowledged functionality in deferrable mode for PubSubPullSensor [airflow]

2024-05-20 Thread via GitHub
eladkal commented on PR #39711: URL: https://github.com/apache/airflow/pull/39711#issuecomment-2120177756 > It is needed to fix the issue in acknowledged functionality in deferrable mode. If this is a bug fix then we should have a unit test to avoid regression -- This is an

[PR] Fix acknowledged functionality in deferrable mode for PubSubPullSensor [airflow]

2024-05-20 Thread via GitHub
MaksYermak opened a new pull request, #39711: URL: https://github.com/apache/airflow/pull/39711 In this PR I have refactored the trigger for PubSubPullSensor. It is needed to fix the issue in acknowledged functionality in deferrable mode. We need to acknowledge after the

Re: [PR] Fix BigQueryCursor execute method if the location is missing [airflow]

2024-05-20 Thread via GitHub
VladaZakharova commented on PR #39659: URL: https://github.com/apache/airflow/pull/39659#issuecomment-2120148328 @eladkal Yes, thank you , will be good to add system test here to cover this scenario: f.e. try to create dataset and table with location different from `US` and then run

Re: [PR] Avoid littering postgres server logs with "could not obtain lock" with HA schedulers [airflow]

2024-05-20 Thread via GitHub
fbertos commented on PR #19842: URL: https://github.com/apache/airflow/pull/19842#issuecomment-2120101929 Dear all, Looks like this is happening again as we are receiving the following messages after upgrading to version Airflow 2.9.1 (we are using PostgreSQL 15.7.0):

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-20 Thread via GitHub
raphaelauv commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1606529029 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] removing deprecation postgres [airflow]

2024-05-20 Thread via GitHub
Taragolis commented on PR #39706: URL: https://github.com/apache/airflow/pull/39706#issuecomment-2120042506 Looks really nice! I would suggest to keep information about PostgresOperator into the documentation, e.g. in warning block This might help users who try to search

Re: [I] KPO - get_logs - logs missing [airflow]

2024-05-20 Thread via GitHub
Taragolis commented on issue #39682: URL: https://github.com/apache/airflow/issues/39682#issuecomment-2120013361 I guess it similar to https://github.com/apache/airflow/issues/39686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] remove deprecations aws eks [airflow]

2024-05-20 Thread via GitHub
Bowrna commented on PR #39709: URL: https://github.com/apache/airflow/pull/39709#issuecomment-212723 I will add the fix for ECS and EMR in this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] remove deprecations aws eks [airflow]

2024-05-20 Thread via GitHub
Bowrna opened a new pull request, #39709: URL: https://github.com/apache/airflow/pull/39709 Related: #39485 --- **^ Add meaningful description above** Read the **[Pull Request

  1   2   3   4   5   6   7   8   9   10   >