[GitHub] [airflow] bkossakowska opened a new pull request, #29735: Add deferrable mode to BigQueryTablePartitionExistenceSensor.

2023-02-23 Thread via GitHub
bkossakowska opened a new pull request, #29735: URL: https://github.com/apache/airflow/pull/29735 Add deferrable mode to BigQueryTablePartitionExistenceSensor. --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] aru-trackunit commented on pull request #29694: Fixing log message + adding more verbose documetation

2023-02-23 Thread via GitHub
aru-trackunit commented on PR #29694: URL: https://github.com/apache/airflow/pull/29694#issuecomment-1442981185 I am not sure who should review it and no one has been attached, @eladkal -> If I made a mistake tagging you please assign to a review someone that could do that -- This is an

[GitHub] [airflow] JammUtkarsh commented on issue #29305: Migrate remaining tests to `pytest`

2023-02-23 Thread via GitHub
JammUtkarsh commented on issue #29305: URL: https://github.com/apache/airflow/issues/29305#issuecomment-1442898723 Hey @Taragolis, If I were to contribute to this issue, would I require the access to cloud environment like GCP or AWS? ref: Phase 1/Provider -- This is an automated

[GitHub] [airflow] github-actions[bot] commented on issue #18962: Please mention AirFlow wordings in documentation & add "Exec specific DAG" role example

2023-02-23 Thread via GitHub
github-actions[bot] commented on issue #18962: URL: https://github.com/apache/airflow/issues/18962#issuecomment-1442889744 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

[GitHub] [airflow] pankajastro commented on pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-23 Thread via GitHub
pankajastro commented on PR #28850: URL: https://github.com/apache/airflow/pull/28850#issuecomment-1442883280 Requesting re-review on this. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] JJJzheng commented on issue #20308: Multi Schedulers don't synchronize task instance state when using Triggerer

2023-02-23 Thread via GitHub
JJJzheng commented on issue #20308: URL: https://github.com/apache/airflow/issues/20308#issuecomment-1442850816 Hi all I met the same problem. And my airflow version is airflow-2.2.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[airflow] branch constraints-main updated: Updating constraints. Github run id:4259455703

2023-02-23 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 4d4ebf0075 Updating

[GitHub] [airflow] pankajastro commented on a diff in pull request #29462: Add deferrable mode for S3ToGCSOperator

2023-02-23 Thread via GitHub
pankajastro commented on code in PR #29462: URL: https://github.com/apache/airflow/pull/29462#discussion_r1116213336 ## airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py: ## @@ -490,3 +494,54 @@ def operations_contain_expected_statuses(

[GitHub] [airflow] uranusjr commented on pull request #29608: Enable passing --xcom-args to tasks test CLI command

2023-02-23 Thread via GitHub
uranusjr commented on PR #29608: URL: https://github.com/apache/airflow/pull/29608#issuecomment-1442811804 I think your plan makes sense. So we’ll use this PR to implement value-passing, and add path support in a later PR. -- This is an automated message from the Apache Git Service. To

[airflow] branch main updated: Update docs re: template_fields typing and subclasses (#29725)

2023-02-23 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 28573c94ff Update docs re: template_fields

[GitHub] [airflow] uranusjr merged pull request #29725: Update docs re: template_fields typing and subclasses

2023-02-23 Thread via GitHub
uranusjr merged PR #29725: URL: https://github.com/apache/airflow/pull/29725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] leozitor commented on issue #11218: Connection Extra Field is not encrypted/masked on editing

2023-02-23 Thread via GitHub
leozitor commented on issue #11218: URL: https://github.com/apache/airflow/issues/11218#issuecomment-1442770740 This issue was not fixed yet? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] LMnet commented on pull request #29434: Impovements for RedshiftDataOperator

2023-02-23 Thread via GitHub
LMnet commented on PR #29434: URL: https://github.com/apache/airflow/pull/29434#issuecomment-1442768784 I rebased and squashed my changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] LMnet commented on a diff in pull request #29434: Impovements for RedshiftDataOperator

2023-02-23 Thread via GitHub
LMnet commented on code in PR #29434: URL: https://github.com/apache/airflow/pull/29434#discussion_r1116478361 ## tests/providers/amazon/aws/operators/test_redshift_data.py: ## @@ -140,3 +140,39 @@ def test_batch_execute(self, mock_conn):

[GitHub] [airflow] LMnet commented on a diff in pull request #29434: Impovements for RedshiftDataOperator

2023-02-23 Thread via GitHub
LMnet commented on code in PR #29434: URL: https://github.com/apache/airflow/pull/29434#discussion_r1116478006 ## airflow/providers/amazon/aws/operators/redshift_data.py: ## @@ -25,6 +26,8 @@ from airflow.providers.amazon.aws.hooks.redshift_data import RedshiftDataHook from

[GitHub] [airflow] josh-fell commented on a diff in pull request #29725: Update docs re: template_fields typing and augmenting with subclasses

2023-02-23 Thread via GitHub
josh-fell commented on code in PR #29725: URL: https://github.com/apache/airflow/pull/29725#discussion_r1116475079 ## docs/apache-airflow/core-concepts/operators.rst: ## @@ -109,7 +109,8 @@ You can also use Jinja templating with nested fields, as long as these nested fi )

[GitHub] [airflow] hussein-awala commented on issue #29582: Cannot trigger the same dag_id within the same second

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29582: URL: https://github.com/apache/airflow/issues/29582#issuecomment-1442651551 Fixed in #27640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] hussein-awala closed issue #29582: Cannot trigger the same dag_id within the same second

2023-02-23 Thread via GitHub
hussein-awala closed issue #29582: Cannot trigger the same dag_id within the same second URL: https://github.com/apache/airflow/issues/29582 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] hussein-awala commented on issue #29582: Cannot trigger the same dag_id within the same second

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29582: URL: https://github.com/apache/airflow/issues/29582#issuecomment-1442650058 Airflow `dags.trigger` removes the microseconds by default, #27640 added a new argument `--no-replace-microseconds` to support keeping them, unfortunately this feature will

[GitHub] [airflow] hussein-awala commented on issue #29582: Cannot trigger the same dag_id within the same second

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29582: URL: https://github.com/apache/airflow/issues/29582#issuecomment-1442631364 Airflow `dags.trigger` removes the microseconds by default, if you want to keep them, you can add the argument `--no-replace-microseconds`. Can you test with this argument?

[GitHub] [airflow] hussein-awala commented on issue #29654: Not able to specify redis as a result backend

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29654: URL: https://github.com/apache/airflow/issues/29654#issuecomment-1442614738 Hello Arthur, > Error: Failed to render chart: exit status 1: Error: values don't meet the specifications of the schema(s) in the following chart(s): airflow:

[GitHub] [airflow] github-actions[bot] commented on pull request #28757: Initial draft of composable timetables

2023-02-23 Thread via GitHub
github-actions[bot] commented on PR #28757: URL: https://github.com/apache/airflow/pull/28757#issuecomment-1442605069 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] eladkal commented on pull request #27638: Add Cloud Run Operator

2023-02-23 Thread via GitHub
eladkal commented on PR #27638: URL: https://github.com/apache/airflow/pull/27638#issuecomment-1442598750 Tests are failing. Could you please fix it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] vemikhaylov commented on pull request #29608: Enable passing --xcom-args to tasks test CLI command

2023-02-23 Thread via GitHub
vemikhaylov commented on PR #29608: URL: https://github.com/apache/airflow/pull/29608#issuecomment-1442592589 > `--xcoms` I love it! Though technically the "task" declaration operates with the `XComArg` layer, inside the task execution it may call `xcom_pull` and `xcom_push`

[GitHub] [airflow] hussein-awala commented on issue #29663: Option to Disable High Cardinality Metrics on Statsd

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29663: URL: https://github.com/apache/airflow/issues/29663#issuecomment-1442583976 I have checked with DataDog, and as I mentioned in my [PR](https://github.com/apache/airflow/pull/28961#issue-1533973742), activating this option will increase the cost, but

[GitHub] [airflow] scastria commented on issue #29582: Cannot trigger the same dag_id within the same second

2023-02-23 Thread via GitHub
scastria commented on issue #29582: URL: https://github.com/apache/airflow/issues/29582#issuecomment-1442577805 We are using MWAA managed Airflow in AWS. They do not directly expose the Airflow REST API, but instead have their own REST API that, I think, calls the Airflow CLI??

[GitHub] [airflow] hussein-awala commented on issue #29666: Multiple Mount Points for Hashicorp Vault Back-end

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29666: URL: https://github.com/apache/airflow/issues/29666#issuecomment-1442565384 Looks like a nice improvement for the Vault backend secret, can you take a look at #29734? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] hussein-awala opened a new pull request, #29734: Support multiple mount points in Vault backend secret

2023-02-23 Thread via GitHub
hussein-awala opened a new pull request, #29734: URL: https://github.com/apache/airflow/pull/29734 closes: #29666 --- Currently we can setup Valut backend secret with a single mount point, and we cannot read variables/secrets from outside this mount point. This PR allows

[GitHub] [airflow-site] jedcunningham merged pull request #742: Revert "Remove old versions of Airflow to save space during builds (#740)"

2023-02-23 Thread via GitHub
jedcunningham merged PR #742: URL: https://github.com/apache/airflow-site/pull/742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow-site] branch gh-pages updated: Deploying to gh-pages from @ 9cd34545f08296d561289be2f20e3cb3cef6e388 

2023-02-23 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/gh-pages by this push: new d45392cf33 Deploying to gh-pages

[GitHub] [airflow-site] jedcunningham opened a new pull request, #742: Revert "Remove old versions of Airflow to save space during builds (#740)"

2023-02-23 Thread via GitHub
jedcunningham opened a new pull request, #742: URL: https://github.com/apache/airflow-site/pull/742 This reverts commit 5c95b627c29520d0e29c6370eb57a9778946cbdc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[airflow-site] branch main updated: Reduce runner disk usage (#741)

2023-02-23 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new 9cd34545f0 Reduce runner disk usage

[GitHub] [airflow-site] jedcunningham merged pull request #741: Reduce runner disk usage

2023-02-23 Thread via GitHub
jedcunningham merged PR #741: URL: https://github.com/apache/airflow-site/pull/741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow-site] dstandish commented on pull request #741: Reduce runner disk usage

2023-02-23 Thread via GitHub
dstandish commented on PR #741: URL: https://github.com/apache/airflow-site/pull/741#issuecomment-1442544635 nice -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow-site] jedcunningham opened a new pull request, #741: Reduce runner disk usage

2023-02-23 Thread via GitHub
jedcunningham opened a new pull request, #741: URL: https://github.com/apache/airflow-site/pull/741 This frees up additional space in the GHA runner, giving us roughly 10Gb of extra space. This is enough to continue building docs in the short term while we figure out an archiving strategy.

[GitHub] [airflow] thirtyseven commented on a diff in pull request #29732: Loop through paginated response to check for cluster id

2023-02-23 Thread via GitHub
thirtyseven commented on code in PR #29732: URL: https://github.com/apache/airflow/pull/29732#discussion_r1116320813 ## airflow/providers/amazon/aws/hooks/emr.py: ## @@ -68,22 +68,24 @@ def get_cluster_id_by_name(self, emr_cluster_name: str, cluster_states: list[str

[GitHub] [airflow] thirtyseven commented on a diff in pull request #29732: Loop through paginated response to check for cluster id

2023-02-23 Thread via GitHub
thirtyseven commented on code in PR #29732: URL: https://github.com/apache/airflow/pull/29732#discussion_r1116320465 ## airflow/providers/amazon/aws/hooks/emr.py: ## @@ -68,22 +68,24 @@ def get_cluster_id_by_name(self, emr_cluster_name: str, cluster_states: list[str

[GitHub] [airflow] eladkal commented on pull request #27638: Add Cloud Run Operator

2023-02-23 Thread via GitHub
eladkal commented on PR #27638: URL: https://github.com/apache/airflow/pull/27638#issuecomment-1442511778 @matthieuvilatte operators usually design to do a specific action thus we like/prefer when operator has a verb in it's name indicate the action for example: `SqlExecuteOperator`,

[airflow] branch main updated: Remove Run task action from UI (#29706)

2023-02-23 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 101d59c4b8 Remove Run task action from UI

[GitHub] [airflow] bbovenzi merged pull request #29706: Remove Run task action from UI

2023-02-23 Thread via GitHub
bbovenzi merged PR #29706: URL: https://github.com/apache/airflow/pull/29706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters

2023-02-23 Thread via GitHub
eladkal commented on PR #29580: URL: https://github.com/apache/airflow/pull/29580#issuecomment-1442507073 Should we modify a bit the description of https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/secrets-backend/index.html#search-path ?

[airflow] branch main updated (5e006d743d -> 7af60d1a10)

2023-02-23 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 5e006d743d Avoid emitting fallback message for S3TaskHandler if streaming logs (#29708) add 7af60d1a10 Improve

[GitHub] [airflow] bbovenzi merged pull request #29563: Improve grid view height calculations

2023-02-23 Thread via GitHub
bbovenzi merged PR #29563: URL: https://github.com/apache/airflow/pull/29563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jedcunningham commented on pull request #29706: Remove Run task action from UI

2023-02-23 Thread via GitHub
jedcunningham commented on PR #29706: URL: https://github.com/apache/airflow/pull/29706#issuecomment-1442501684 Yeah, when troubleshooting something else today, I happened to find that --local ends up using --raw as well . Either way, I think we should address the UI and CLI separately.

[GitHub] [airflow] hussein-awala commented on issue #29733: Databricks create/reset then run-now

2023-02-23 Thread via GitHub
hussein-awala commented on issue #29733: URL: https://github.com/apache/airflow/issues/29733#issuecomment-1442465033 Sounds like a good feature, you can create a new operator `DatabricksCreateJobOperator` with a boolean argument `reset` to specify if we want to create the job or overwrite

[GitHub] [airflow] Taragolis commented on pull request #29732: Loop through paginated response to check for cluster id

2023-02-23 Thread via GitHub
Taragolis commented on PR #29732: URL: https://github.com/apache/airflow/pull/29732#issuecomment-1442456791 > I took a stab at this but I am at a loss as to how to test without spinning up 51 EMR clusters myself. If anyone has a suggestion for how to whitebox, mock, or test this, happy to

[GitHub] [airflow] dstandish merged pull request #29708: Avoid including fallback message for S3TaskHandler if streaming logs

2023-02-23 Thread via GitHub
dstandish merged PR #29708: URL: https://github.com/apache/airflow/pull/29708 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] dstandish closed issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3

2023-02-23 Thread via GitHub
dstandish closed issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3 URL: https://github.com/apache/airflow/issues/29393 -- This is an automated message from the Apache Git Service. To respond to the message, please

[airflow] branch main updated (13098d5c35 -> 5e006d743d)

2023-02-23 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 13098d5c35 reorient grid buttons (#29564) add 5e006d743d Avoid emitting fallback message for S3TaskHandler if

[airflow] branch main updated (09654eee33 -> 13098d5c35)

2023-02-23 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 09654eee33 some fixes in the issue triage process doc (#29730) add 13098d5c35 reorient grid buttons (#29564)

[GitHub] [airflow] bbovenzi merged pull request #29564: reorient grid buttons

2023-02-23 Thread via GitHub
bbovenzi merged PR #29564: URL: https://github.com/apache/airflow/pull/29564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] kyle-winkelman commented on issue #29733: Databricks create/reset then run-now

2023-02-23 Thread via GitHub
kyle-winkelman commented on issue #29733: URL: https://github.com/apache/airflow/issues/29733#issuecomment-1442445629 I have come up with a work around that uses the SimpleHttpOperator to directly hit the `api/2.1/jobs/create` (or `api/2.1/jobs/reset`) endpoint and then XComs the `job_id`

[airflow] branch main updated: some fixes in the issue triage process doc (#29730)

2023-02-23 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 09654eee33 some fixes in the issue triage process

[GitHub] [airflow] eladkal merged pull request #29730: some fixes in the issue triage process doc

2023-02-23 Thread via GitHub
eladkal merged PR #29730: URL: https://github.com/apache/airflow/pull/29730 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] subuserzero commented on issue #29366: default_args feature incompatible with Dynamic Task Mapping

2023-02-23 Thread via GitHub
subuserzero commented on issue #29366: URL: https://github.com/apache/airflow/issues/29366#issuecomment-1442444585 > **As a quick potential workaround**, you may wrap a mapped operator into a `TaskGroup` with `default_args={}`, because here is what [docs

[GitHub] [airflow] Taragolis commented on a diff in pull request #29602: Change config max tis per query

2023-02-23 Thread via GitHub
Taragolis commented on code in PR #29602: URL: https://github.com/apache/airflow/pull/29602#discussion_r1116254099 ## airflow/config_templates/default_airflow.cfg: ## @@ -1120,11 +1120,12 @@ catchup_by_default = True ignore_first_depends_on_past_by_default = True # This

[GitHub] [airflow] Taragolis commented on pull request #29511: Add Landing Times entry to UI docs

2023-02-23 Thread via GitHub
Taragolis commented on PR #29511: URL: https://github.com/apache/airflow/pull/29511#issuecomment-1442438064 > Sorry to have missed the checks. Should be good now @Taragolis That was just friendly reminder, sometimes contributors forget about this parts (static checks) and how to

[GitHub] [airflow] kyle-winkelman commented on issue #29733: Databricks create/reset then run-now

2023-02-23 Thread via GitHub
kyle-winkelman commented on issue #29733: URL: https://github.com/apache/airflow/issues/29733#issuecomment-1442437525 I have spoken to my Databricks representative and they don't have a high priority on syncing up the api for /2.1/jobs/runs/submit with all the options available in

[GitHub] [airflow] Taragolis commented on a diff in pull request #29522: Add support in AWS Batch Operator for multinode jobs

2023-02-23 Thread via GitHub
Taragolis commented on code in PR #29522: URL: https://github.com/apache/airflow/pull/29522#discussion_r1116243761 ## airflow/providers/amazon/aws/operators/batch.py: ## @@ -174,18 +209,27 @@ def submit_job(self, context: Context): self.job_definition,

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29733: Databricks create/reset then run-now

2023-02-23 Thread via GitHub
boring-cyborg[bot] commented on issue #29733: URL: https://github.com/apache/airflow/issues/29733#issuecomment-1442426125 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] kyle-winkelman opened a new issue, #29733: Databricks create/reset then run-now

2023-02-23 Thread via GitHub
kyle-winkelman opened a new issue, #29733: URL: https://github.com/apache/airflow/issues/29733 ### Description Allow an Airflow DAG to define a Databricks job with the /2.1/jobs/create (or /2.1/jobs/reset) endpoint then run that same job with the /2.1/jobs/run-now endpoint. This

[GitHub] [airflow] nsAstro commented on issue #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination

2023-02-23 Thread via GitHub
nsAstro commented on issue #29712: URL: https://github.com/apache/airflow/issues/29712#issuecomment-1442425890 I took a stab at this but I am at a loss as to how to test without spinning up 51 EMR clusters myself. If anyone has a suggestion for how to whitebox, or test this, happy to

[GitHub] [airflow] neko1437 commented on pull request #23618: Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

2023-02-23 Thread via GitHub
neko1437 commented on PR #23618: URL: https://github.com/apache/airflow/pull/23618#issuecomment-1442424309 @potiuk > What make you think so? Because at the top of [#23497](https://github.com/apache/airflow/issues/23497) it says `Fixed by #23618 · May be fixed by #28336`. So

[GitHub] [airflow] nsAstro opened a new pull request, #29732: Loop through paginated response to check for cluster id

2023-02-23 Thread via GitHub
nsAstro opened a new pull request, #29732: URL: https://github.com/apache/airflow/pull/29732 Loops through the paginated emr list_clusters endpoint and searches each set for the given cluster_id. closes: #29712 -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] Taragolis closed issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-23 Thread via GitHub
Taragolis closed issue #29640: NoBoundaryInMultipartDefect raised using S3Hook URL: https://github.com/apache/airflow/issues/29640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] Taragolis commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-23 Thread via GitHub
Taragolis commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1442417038 Let's close it. Just couple of comments Even it it not a problem with airflow, but I think we need to remove part with check if key exists, from this method in S3Hook. The

[GitHub] [airflow] hussein-awala opened a new pull request, #29730: some fixes in the issue triage process doc

2023-02-23 Thread via GitHub
hussein-awala opened a new pull request, #29730: URL: https://github.com/apache/airflow/pull/29730 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for

[GitHub] [airflow] collinmcnulty commented on pull request #28943: Add handling logic on CeleryExecutor to reschedule task stuck in queued status

2023-02-23 Thread via GitHub
collinmcnulty commented on PR #28943: URL: https://github.com/apache/airflow/pull/28943#issuecomment-1442398497 @tanelk It is possible for tasks to be stuck in queued for reasons other than those checked in `stalled_task_timeout`. This seems like it would be more general purpose. See

[GitHub] [airflow] potiuk commented on pull request #23618: Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

2023-02-23 Thread via GitHub
potiuk commented on PR #23618: URL: https://github.com/apache/airflow/pull/23618#issuecomment-1442393871 What make you think so? and why does it matter? Why do you ask that question and what are you trying to achieve? -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] neko1437 commented on pull request #23618: Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

2023-02-23 Thread via GitHub
neko1437 commented on PR #23618: URL: https://github.com/apache/airflow/pull/23618#issuecomment-1442390913 @potiuk, isn't this a duplicate of [#28336](https://github.com/apache/airflow/pull/28336) and looks like it is pending a review? -- This is an automated message from the Apache Git

[GitHub] [airflow] ferruzzi commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-23 Thread via GitHub
ferruzzi commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1442380228 I'll poke it, maybe I can make a contribution to botocore to get the warning gone, but I am inclined to agree that it is not an Airflow issue. I'll have a look at it, likely not

[GitHub] [airflow] kaxil commented on a diff in pull request #29721: Make serde lazy-load dependent modules

2023-02-23 Thread via GitHub
kaxil commented on code in PR #29721: URL: https://github.com/apache/airflow/pull/29721#discussion_r1116203925 ## airflow/serialization/serde.py: ## @@ -270,38 +273,94 @@ def _stringify(classname: str, version: int, value: T | None) -> str: return s -def _register():

[GitHub] [airflow] antonio-antuan commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-23 Thread via GitHub
antonio-antuan commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1442376775 Looks like that. Anyway it looks like the issue is not related to airflow, so feel free to close it if you want :) -- This is an automated message from the Apache Git

[GitHub] [airflow] ferruzzi commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-23 Thread via GitHub
ferruzzi commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1442374253 If I understand correctly: everything is working fine, but it's printing an unexpected error message in the logs? Is that accurate? -- This is an automated message from the

[GitHub] [airflow] kaxil commented on a diff in pull request #29462: Add deferrable mode for S3ToGCSOperator

2023-02-23 Thread via GitHub
kaxil commented on code in PR #29462: URL: https://github.com/apache/airflow/pull/29462#discussion_r1116199803 ## generated/provider_dependencies.json: ## @@ -359,6 +359,7 @@ "google-cloud-secret-manager>=0.2.0,<2.0.0", "google-cloud-spanner>=1.10.0,<2.0.0",

[GitHub] [airflow] kaxil commented on a diff in pull request #29462: Add deferrable mode for S3ToGCSOperator

2023-02-23 Thread via GitHub
kaxil commented on code in PR #29462: URL: https://github.com/apache/airflow/pull/29462#discussion_r1116199099 ## docs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst: ## @@ -37,6 +37,16 @@ to transfer data from Amazon S3 to Google Cloud Storage.

[GitHub] [airflow] kaxil commented on a diff in pull request #29462: Add deferrable mode for S3ToGCSOperator

2023-02-23 Thread via GitHub
kaxil commented on code in PR #29462: URL: https://github.com/apache/airflow/pull/29462#discussion_r1116198006 ## airflow/providers/google/cloud/transfers/s3_to_gcs.py: ## @@ -104,6 +127,7 @@ class S3ToGCSOperator(S3ListOperator): "google_impersonation_chain", )

[GitHub] [airflow] dimberman commented on a diff in pull request #29522: Add support in AWS Batch Operator for multinode jobs

2023-02-23 Thread via GitHub
dimberman commented on code in PR #29522: URL: https://github.com/apache/airflow/pull/29522#discussion_r1116193774 ## airflow/providers/amazon/aws/operators/batch.py: ## @@ -174,18 +209,27 @@ def submit_job(self, context: Context): self.job_definition,

[GitHub] [airflow] john-jac commented on pull request #29511: Add Landing Times entry to UI docs

2023-02-23 Thread via GitHub
john-jac commented on PR #29511: URL: https://github.com/apache/airflow/pull/29511#issuecomment-1442365714 > @john-jac could you fix Static Checks, the only way in your case it is use `pre-commit`, see [Static code

[GitHub] [airflow] dimberman commented on a diff in pull request #29602: Change config max tis per query

2023-02-23 Thread via GitHub
dimberman commented on code in PR #29602: URL: https://github.com/apache/airflow/pull/29602#discussion_r1116186682 ## airflow/config_templates/default_airflow.cfg: ## @@ -1120,11 +1120,12 @@ catchup_by_default = True ignore_first_depends_on_past_by_default = True # This

[GitHub] [airflow] dimberman commented on pull request #29602: Change config max tis per query

2023-02-23 Thread via GitHub
dimberman commented on PR #29602: URL: https://github.com/apache/airflow/pull/29602#issuecomment-1442354625 Hi @lyso not a big deal but in the future please don't put your name in the PR title. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] dimberman commented on a diff in pull request #29711: fix update_mask in patch variable route

2023-02-23 Thread via GitHub
dimberman commented on code in PR #29711: URL: https://github.com/apache/airflow/pull/29711#discussion_r1116182339 ## airflow/api_connexion/endpoints/variable_endpoint.py: ## @@ -103,15 +109,22 @@ def patch_variable(*, variable_key: str, update_mask: UpdateMask = None) -> Resp

[GitHub] [airflow] bugraoz93 commented on issue #29312: AIP-44 Migrate TaskInstance._run_raw_task to Internal API

2023-02-23 Thread via GitHub
bugraoz93 commented on issue #29312: URL: https://github.com/apache/airflow/issues/29312#issuecomment-1442338302 Thanks for pointing that out mhenc! No pressure potiuk, looking forward to seeing it :) -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] potiuk commented on pull request #23618: Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

2023-02-23 Thread via GitHub
potiuk commented on PR #23618: URL: https://github.com/apache/airflow/pull/23618#issuecomment-1442302100 Apparently It's been reverted and author attempted to fix it https://github.com/apache/airflow/pull/23658 and since author "abandoned" it, it's been closed. But if you want to pick it

[GitHub] [airflow] potiuk commented on pull request #29728: Set upper limit for `PyGithub` package

2023-02-23 Thread via GitHub
potiuk commented on PR #29728: URL: https://github.com/apache/airflow/pull/29728#issuecomment-1442296898 It looks like - from your issue @josh-fell that they are eyeying fixing it. So I would rather use != 1.58 now. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] josh-fell commented on a diff in pull request #29725: Update docs re: template_fields typing and augmenting with subclasses

2023-02-23 Thread via GitHub
josh-fell commented on code in PR #29725: URL: https://github.com/apache/airflow/pull/29725#discussion_r1116133821 ## docs/apache-airflow/core-concepts/operators.rst: ## @@ -109,7 +109,8 @@ You can also use Jinja templating with nested fields, as long as these nested fi )

[GitHub] [airflow] potiuk commented on issue #29389: Dag with kubernetes executor are being killed with "PID of job runner does not match"

2023-02-23 Thread via GitHub
potiuk commented on issue #29389: URL: https://github.com/apache/airflow/issues/29389#issuecomment-1442293373 I have no idea, but I doubt. I added "needs-triage" and possibly someone will take a look at it. I think adding milestone was just a mistake. -- This is an automated message from

[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test

2023-02-23 Thread via GitHub
potiuk commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1442289253 Yeah. I noticed it too :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] uranusjr commented on a diff in pull request #29725: Update docs re: template_fields typing and augmenting with subclasses

2023-02-23 Thread via GitHub
uranusjr commented on code in PR #29725: URL: https://github.com/apache/airflow/pull/29725#discussion_r1116128192 ## docs/apache-airflow/core-concepts/operators.rst: ## @@ -109,7 +109,8 @@ You can also use Jinja templating with nested fields, as long as these nested fi )

[GitHub] [airflow] potiuk commented on pull request #29725: Update docs re: template_fields typing and augmenting with subclasses

2023-02-23 Thread via GitHub
potiuk commented on PR #29725: URL: https://github.com/apache/airflow/pull/29725#issuecomment-1442283233 Very common question answered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ferruzzi commented on pull request #29347: SSH Provider: Add cmd_timeout to ssh connection extra

2023-02-23 Thread via GitHub
ferruzzi commented on PR #29347: URL: https://github.com/apache/airflow/pull/29347#issuecomment-1442274063 @Taragolis Thanks for picking up the review on this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] uranusjr commented on pull request #29706: Remove Run task action from UI

2023-02-23 Thread via GitHub
uranusjr commented on PR #29706: URL: https://github.com/apache/airflow/pull/29706#issuecomment-1442270187 IIRC `--raw` is needed for (some?) other executors to run the task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] uranusjr commented on pull request #29406: Add Fail Fast feature for DAGs

2023-02-23 Thread via GitHub
uranusjr commented on PR #29406: URL: https://github.com/apache/airflow/pull/29406#issuecomment-1442267128 How should this interact with trigger rules? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] uranusjr commented on pull request #29714: Add database check_migrations config

2023-02-23 Thread via GitHub
uranusjr commented on PR #29714: URL: https://github.com/apache/airflow/pull/29714#issuecomment-1442259490 > Are there any existing Airflow benchmarks related to applying migrations? I need more information to make a call on this - it would be great if we could have some data to make this

[GitHub] [airflow] uranusjr commented on pull request #29723: Don't use importlib.metadata to get Version for speed

2023-02-23 Thread via GitHub
uranusjr commented on PR #29723: URL: https://github.com/apache/airflow/pull/29723#issuecomment-1442254077 There are some more references to `version` that need to be changed ``` scripts/ci/pre_commit/pre_commit_update_versions.py:31: error: Module "setup" has no attribute

[GitHub] [airflow] ferruzzi commented on issue #12771: Instrument Airflow with Opentracing/Opentelemetry

2023-02-23 Thread via GitHub
ferruzzi commented on issue #12771: URL: https://github.com/apache/airflow/issues/12771#issuecomment-1442250162 Thanks for that update, sorry I didn't do it myself. :+1: Current state, if anyone cares: I have a Breeze dev environment that launches Otel-Collector,

[GitHub] [airflow] bbovenzi commented on a diff in pull request #29413: Add a new graph inside of the grid view

2023-02-23 Thread via GitHub
bbovenzi commented on code in PR #29413: URL: https://github.com/apache/airflow/pull/29413#discussion_r1116075518 ## airflow/www/static/js/dag/details/graph/Node.tsx: ## @@ -0,0 +1,156 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [airflow] bbovenzi commented on a diff in pull request #29413: Add a new graph inside of the grid view

2023-02-23 Thread via GitHub
bbovenzi commented on code in PR #29413: URL: https://github.com/apache/airflow/pull/29413#discussion_r1116070096 ## airflow/www/static/js/dag/details/graph/utils.ts: ## @@ -0,0 +1,116 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[airflow] branch revert-29408-docker-compose-change-example updated (793be818fc -> 026f59708d)

2023-02-23 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git omit 793be818fc Revert "Improve health checks in example docker-compose and

[GitHub] [airflow] uranusjr commented on a diff in pull request #29721: Make serde lazy-load dependent modules

2023-02-23 Thread via GitHub
uranusjr commented on code in PR #29721: URL: https://github.com/apache/airflow/pull/29721#discussion_r1116057525 ## airflow/serialization/serde.py: ## @@ -270,38 +273,94 @@ def _stringify(classname: str, version: int, value: T | None) -> str: return s -def

  1   2   >