date:20230131

[GitHub] [airflow-client-go] zbysir opened a new issue, #38: Request.Execute() return `undefined response type` error

2023-01-31 Thread via GitHub



zbysir opened a new issue, #38:
URL: https://github.com/apache/airflow-client-go/issues/38

   ```
   ➜  ~ curl -i -X GET 'https://airflow.xxx.com/api/v1/dags/xx/dagRuns/xxx' \
   -H 'Content-Type: application/json' -H 'Accept: application/json' \
   --user "xxx:xxx" \
   -d '{}'
   HTTP/2 404
   date: Wed, 01 Feb 2023 07:42:55 GMT
   content-type: application/problem+json
   content-length: 263
   server: gunicorn
   x-robots-tag: noindex, nofollow
   
   {
 "detail": "DAGRun with DAG ID: 'xxx' and DagRun ID: 'xxx' not found",
 "status": 404,
 "title": "DAGRun not found",
 "type": 
"https://airflow.apache.org/docs/apache-airflow/2.5.0/stable-rest-api-ref.html#section/Errors/NotFound";
   }
   ```
   
   In airflow version 2.5.0, Content-type header in error response is 
`application/problem+json`, but in this repo the json content-type is check by 
regexp 
   ```
   jsonCheck = 
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)
   ```
   
   The regexp can't mach `application/problem+json`, I guess that's the problem.
   
   > About 
[problem+json](https://datatracker.ietf.org/doc/html/draft-nottingham-http-problem-07)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #27758: Enable trigger logging in webserver

2023-01-31 Thread via GitHub



Taragolis commented on code in PR #27758:
URL: https://github.com/apache/airflow/pull/27758#discussion_r1092857724


##
airflow/jobs/local_task_job.py:
##
@@ -203,6 +205,7 @@ def sigusr2_debug_handler(signum, frame):
 )
 finally:
 self.on_kill()
+return return_code

Review Comment:
   I guess this return might be the reason why Segmentation Fault handler tests 
broken, and potentially others like SIGTERM (but their not raise any error or 
run in separate process).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Bhawana-Barfa opened a new issue, #29281: (psycopg2.errors.FeatureNotSupported) This ALTER TABLE command is not yet supported .While deploying airflow with Postgres

2023-01-31 Thread via GitHub



Bhawana-Barfa opened a new issue, #29281:
URL: https://github.com/apache/airflow/issues/29281

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Airflow Version : 2.2.3 
   I am trying to deploy airflow with postgres on cluster with helm chart . 
When I install helm , upgrade-db goes into Error with the log error 
sqlalchemy.exc.NotSupportedError: (psycopg2.errors.FeatureNotSupported) This 
ALTER TABLE command is not yet supported.
   
[airflow_err.txt](https://github.com/apache/airflow/files/10554275/airflow_err.txt)
   Which is coming from 
https://github.com/apache/airflow/blob/main/airflow/migrations/versions/0008_1_6_0_task_duration.py
   https://user-images.githubusercontent.com/124127106/215976314-5755f151-11ba-4778-8afd-a2615420afc2.png";>
   
   Please let me know if Im doing anything wrong here, or suggest what need to 
done here. 
   
   
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   vi values.yaml
   externalDatabase:
   type: postgres
   Insert postgres host port user and secret.
   
   
   helm install -f values.yaml -n airflow my-airflow-cluster .
   
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29281: (psycopg2.errors.FeatureNotSupported) This ALTER TABLE command is not yet supported .While deploying airflow with Postgres

2023-01-31 Thread boring-cyborg



boring-cyborg[bot] commented on issue #29281:
URL: https://github.com/apache/airflow/issues/29281#issuecomment-1411585269

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on pull request #28721: Use connection URI in SqliteHook

2023-01-31 Thread via GitHub



feluelle commented on PR #28721:
URL: https://github.com/apache/airflow/pull/28721#issuecomment-1411582283

   Yes, good catch @ashb and @josh-fell thank you for the heads-up. That is a 
tricky one. I would say we merge the other PR first and I will just update my 
PR and docs to reflect the changes i.e. that from now on you could use certain 
fields such as login, password, etc. The current documentation also states that 
the fields are required, which they are not even with my change using the URL.
   
   But I would like to bring @atharvmishra1999 to the discussion. How do you 
think we should handle the collision?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ecerulm commented on a diff in pull request #29016: Fix leak sensitive field via V1EnvVar on exception

2023-01-31 Thread via GitHub



ecerulm commented on code in PR #29016:
URL: https://github.com/apache/airflow/pull/29016#discussion_r1092846149


##
airflow/utils/log/secrets_masker.py:
##
@@ -200,10 +222,18 @@ def _redact(self, item: Redactable, name: str | None, 
depth: int) -> Redacted:
 if name and should_hide_value_for_key(name):
 return self._redact_all(item, depth)
 if isinstance(item, dict):
-return {
+to_return = {
 dict_key: self._redact(subval, name=dict_key, depth=(depth 
+ 1))
 for dict_key, subval in item.items()
 }
+return to_return
+elif isinstance(item, ConvertableToDict):  # things like V1EnvVar

Review Comment:
   @uranusjr , what do you think? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29224: add output format arg for `cli.dags.trigger`

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29224:
URL: https://github.com/apache/airflow/pull/29224#discussion_r1092826281


##
tests/api/client/test_local_client.py:
##
@@ -128,6 +128,45 @@ def test_trigger_dag(self, mock):
 )
 mock.reset_mock()
 
+# test output
+queued_at = pendulum.now()
+started_at = pendulum.now()
+mock.return_value = DagRun(
+dag_id=test_dag_id,
+run_id=run_id,
+queued_at=queued_at,
+execution_date=EXECDATE,
+start_date=started_at,
+external_trigger=True,
+state=DagRunState.QUEUED,
+conf={},
+run_type=DagRunType.MANUAL,
+data_interval=(EXECDATE, EXECDATE + 
pendulum.duration(hours=1)),
+)
+expected_dag_run = {
+"conf": {},
+"dag_id": test_dag_id,
+"dag_run_id": run_id,
+"data_interval_end": EXECDATE,
+"data_interval_start": EXECDATE + pendulum.duration(hours=1),
+"end_date": None,
+"external_trigger": True,
+"last_scheduling_decision": None,
+"logical_date": EXECDATE,
+"run_type": DagRunType.MANUAL,
+"start_date": started_at,
+"state": DagRunState.QUEUED,
+}
+dag_run = self.client.trigger_dag(dag_id=test_dag_id)
+assert expected_dag_run == dag_run
+mock.reset_mock()
+
+# test output when no DagRun is created
+mock.return_value = None
+dag_run = self.client.trigger_dag(dag_id=test_dag_id)
+assert None is dag_run

Review Comment:
   ```suggestion
   assert not dag_run
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Adjust some wording in the release process (#29278)

2023-01-31 Thread ephraimanierobi

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 05fb80ee93 Adjust some wording in the release process (#29278)
05fb80ee93 is described below

commit 05fb80ee9373835b2f72fad3e9976cf29aeca23b
Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com>
AuthorDate: Wed Feb 1 00:50:15 2023 -0600

Adjust some wording in the release process (#29278)

This rewords the header in our testing issue, and fixes the rat
section of the helm vote email.
---
 dev/ISSUE_TEMPLATE.md.jinja2 | 7 ++-
 dev/README_RELEASE_HELM_CHART.md | 4 ++--
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/dev/ISSUE_TEMPLATE.md.jinja2 b/dev/ISSUE_TEMPLATE.md.jinja2
index be8fa5d74c..7cef2cbef4 100644
--- a/dev/ISSUE_TEMPLATE.md.jinja2
+++ b/dev/ISSUE_TEMPLATE.md.jinja2
@@ -17,12 +17,9 @@
   under the License.
 -#}
 
+We are kindly requesting that contributors to [{{link_text}}]({{link}}) help 
test the RC.
 
-We have a kind request for all the contributors to the latest 
[{{link_text}}]({{link}}).
-
-Could you please help us to test the RC versions of Airflow?
-
-Please let us know in the comment if the issue is addressed in the latest RC.
+Please let us know by commenting if the issue is addressed in the latest RC.
 
 {% for pr_number in pr_list %}
 {%- set pr = pull_requests[pr_number] -%}
diff --git a/dev/README_RELEASE_HELM_CHART.md b/dev/README_RELEASE_HELM_CHART.md
index f6ad27fc13..c655072941 100644
--- a/dev/README_RELEASE_HELM_CHART.md
+++ b/dev/README_RELEASE_HELM_CHART.md
@@ -328,11 +328,11 @@ Please vote accordingly:
 Only votes from PMC members are binding, but members of the community are
 encouraged to test the release and vote with "(non-binding)".
 
-For license checks, the .rat-excludes files is included, so you can run the 
following to verify licenses (just update $PATH_TO_RAT):
+For license checks, the .rat-excludes files is included, so you can run the 
following to verify licenses (just update your path to rat):
 
 tar -xvf airflow-chart-${VERSION_WITHOUT_RC}-source.tar.gz
 cd airflow-chart-${VERSION_WITHOUT_RC}
-java -jar $PATH_TO_RAT/apache-rat-0.13/apache-rat-0.13.jar chart -E 
.rat-excludes
+java -jar apache-rat-0.13.jar chart -E .rat-excludes
 
 Please note that the version number excludes the \`rcX\` string, so it's now
 simply ${VERSION_WITHOUT_RC}. This will allow us to rename the artifact 
without modifying

[GitHub] [airflow] ephraimbuddy merged pull request #29278: Adjust some wording in the release process

2023-01-31 Thread via GitHub



ephraimbuddy merged PR #29278:
URL: https://github.com/apache/airflow/pull/29278


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #29093: [WIP] Enable tags for statsd metrics in a backward compatible way

2023-01-31 Thread via GitHub



uranusjr commented on PR #29093:
URL: https://github.com/apache/airflow/pull/29093#issuecomment-1411518422

   If the goal is to remove statsd altogether eventually, emitting an event 
under multiple keys feels like the easiest solution to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] sudeepgupta90 commented on issue #29273: Status of testing of Apache Airflow Helm Chart 1.8.0rc1

2023-01-31 Thread via GitHub



sudeepgupta90 commented on issue #29273:
URL: https://github.com/apache/airflow/issues/29273#issuecomment-1411492577

   Checked #28604  - all good
   
   On Wed, Feb 1, 2023 at 10:46 AM Aakcht ***@***.***> wrote:
   
   > Checked #27544  , #27779
   >  and #29032
   >  - all good.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Aakcht commented on issue #29273: Status of testing of Apache Airflow Helm Chart 1.8.0rc1

2023-01-31 Thread via GitHub



Aakcht commented on issue #29273:
URL: https://github.com/apache/airflow/issues/29273#issuecomment-1411481922

   Checked #27544 , #27779 and #29032 - all good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] riyaj8888 opened a new issue, #29280: elastic-search index creation

2023-01-31 Thread via GitHub



riyaj opened a new issue, #29280:
URL: https://github.com/apache/airflow/issues/29280

   ### Apache Airflow Provider(s)
   
   elasticsearch
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-elasticsearch == 4.3.3
   
   ### Apache Airflow version
   
   2.3.2
   
   ### Operating System
   
   windows
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   in docker-compose i have added elastic-search as service and using that 
service i am trying to connect to elastic cluster and create index in DAG file 
but i am getting following errors
   `new connection refused `
   
   `from airflow.providers.elasticsearch.hooks.elasticsearch import 
ElasticsearchHook
   
   
   from airflow.operators.dummy_operator import DummyOperator
   from datetime import datetime,timedelta
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator
   from elasticsearch import Elasticsearch
   
   
   default_args = {"owner":"default",
   "depend_on_past":False,
   "start_date":datetime(2023, 1 ,26),
   "retries":1,
   "retry_delay":timedelta(minutes=15)}
   
   
   
   mappings = {
   
   "properties": {
   "title": {"type": "text"},
   "description": {"type": "text"},
   "sent-emb": {
   "type": "dense_vector",
   "dims": 768,
   "index": True,
   "similarity": "l2_norm"
   }
   }
   }
   
   def get_activated_sources():
   dense_vector_dim = 768
   index_name = 'test_es'
   es = Elasticsearch("http://localhost:9200";)
   # es.indices.create(index=index_name, body=mappings, ignore=400)
   # es_index(es,dense_vector_dim,index_name)
   # es_hook = ElasticsearchHook(hosts= ['http://localhost:9200'])
   # connection = es_hook.get_conn()
   print(es)
   
   
   with DAG 
("es_dag",default_args=default_args,schedule_interval="@daily",catchup=False) 
as dag:
   
   start_task = DummyOperator(task_id="dummpy_task")
   
   hook_task = 
PythonOperator(task_id="hook_task",python_callable=get_activated_sources)
   
   start_task >> hook_task
   
   
   
   `
   
   I am able to connect to elastic-search cluster ,but when i am trying run 
   es.indices.create(index=index_name, body=mappings, ignore=400) this throwing 
error.
   
   can we create elastic index inside airflow DAG or not?
   
   thanks 
   
   ### What you think should happen instead
   
   same script is able to create elastic index on local machine , but Inside 
DAG script its throwing error.
   
   ### How to reproduce
   
   in docker-compose i have added elastic-search as service and using that 
service i am trying to connect to elastic cluster and create index in DAG file 
but i am getting following errors
   `new connection refused `
   
   `from airflow.providers.elasticsearch.hooks.elasticsearch import 
ElasticsearchHook
   
   
   from airflow.operators.dummy_operator import DummyOperator
   from datetime import datetime,timedelta
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator
   from elasticsearch import Elasticsearch
   
   
   default_args = {"owner":"default",
   "depend_on_past":False,
   "start_date":datetime(2023, 1 ,26),
   "retries":1,
   "retry_delay":timedelta(minutes=15)}
   
   
   
   mappings = {
   
   "properties": {
   "title": {"type": "text"},
   "description": {"type": "text"},
   "sent-emb": {
   "type": "dense_vector",
   "dims": 768,
   "index": True,
   "similarity": "l2_norm"
   }
   }
   }
   
   def get_activated_sources():
   dense_vector_dim = 768
   index_name = 'test_es'
   es = Elasticsearch("http://localhost:9200";)
   # es.indices.create(index=index_name, body=mappings, ignore=400)
   # es_index(es,dense_vector_dim,index_name)
   # es_hook = ElasticsearchHook(hosts= ['http://localhost:9200'])
   # connection = es_hook.get_conn()
   print(es)
   
   
   with DAG 
("es_dag",default_args=default_args,schedule_interval="@daily",catchup=False) 
as dag:
   
   start_task = DummyOperator(task_id="dummpy_task")
   
   hook_task = 
PythonOperator(task_id="hook_task",python_callable=get_activated_sources)
   
   start_task >> hook_task
   
   
   
   `
   
   I am able to connect to elastic-search cluster ,but when i am trying run 
   es.indices.create(index=index_name, body=mappings, ignore=400) this throwing 
error.
   
   can we create elastic index inside airflow DAG or not?
   
   thanks 
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of C

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29280: elastic-search index creation

2023-01-31 Thread boring-cyborg



boring-cyborg[bot] commented on issue #29280:
URL: https://github.com/apache/airflow/issues/29280#issuecomment-1411462214

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] taragurung commented on issue #29272: Airflow Dag fails with no logs , all services looks ok

2023-01-31 Thread via GitHub



taragurung commented on issue #29272:
URL: https://github.com/apache/airflow/issues/29272#issuecomment-1411460251

   This is what I see now
   ```
   [2023-01-31, 20:28:18 UTC] {local_task_job.py:212} WARNING - State of this 
instance has been externally set to skipped. Terminating instance.
   [2023-01-31, 20:28:18 UTC] {process_utils.py:124} INFO - Sending 
Signals.SIGTERM to group 44. PIDs of all processes in the group: [44]
   [2023-01-31, 20:28:18 UTC] {process_utils.py:75} INFO - Sending the signal 
Signals.SIGTERM to group 44
   [2023-01-31, 20:28:19 UTC] {taskinstance.py:1430} ERROR - Received SIGTERM. 
Terminating subprocesses.
   [2023-01-31, 20:28:19 UTC] {process_utils.py:70} INFO - Process 
psutil.Process(pid=44, status='terminated', exitcode=0, started='19:28:24') 
(44) terminated with exit code 0
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell commented on a diff in pull request #29274: log the observed status in redshift sensor

2023-01-31 Thread via GitHub



josh-fell commented on code in PR #29274:
URL: https://github.com/apache/airflow/pull/29274#discussion_r1092751084


##
airflow/providers/amazon/aws/sensors/redshift_cluster.py:
##
@@ -56,8 +56,14 @@ def __init__(
 self.aws_conn_id = aws_conn_id
 
 def poke(self, context: Context):
-self.log.info("Poking for status : %s\nfor cluster %s", 
self.target_status, self.cluster_identifier)
-return self.hook.cluster_status(self.cluster_identifier) == 
self.target_status
+current_status = self.hook.cluster_status(self.cluster_identifier)
+self.log.info(
+"Poked cluster %s for status '%s', found status '%s'",

Review Comment:
   ```suggestion
   "Poked cluster %s for status %r, found status %r",
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham opened a new pull request, #29279: Patch only single label when marking KPO checked

2023-01-31 Thread via GitHub



jedcunningham opened a new pull request, #29279:
URL: https://github.com/apache/airflow/pull/29279

   Instead of sending over a whole pod, we can simply send over the label to 
add to the pod instead. This is less work for us, and also avoids any issues 
like #24015.
   
   Fixes #24015


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] EricGao888 commented on issue #29273: Status of testing of Apache Airflow Helm Chart 1.8.0rc1

2023-01-31 Thread via GitHub



EricGao888 commented on issue #29273:
URL: https://github.com/apache/airflow/issues/29273#issuecomment-1411385945

   I have validated #28858 which fixed some comments in the source code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham opened a new pull request, #29278: Adjust some wording in the release process

2023-01-31 Thread via GitHub



jedcunningham opened a new pull request, #29278:
URL: https://github.com/apache/airflow/pull/29278

   This rewords our the header in our testing issue, and fixes the rat section 
of the helm vote email.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #24825: Dockerfile centos

2023-01-31 Thread github-actions



github-actions[bot] commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1411251315

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29276: AWS system test eks and fargate profile: get subnets as external parameter

2023-01-31 Thread via GitHub



o-nikolas commented on PR #29276:
URL: https://github.com/apache/airflow/pull/29276#issuecomment-1411220093

   FYI: @vincbeck is just making some internal preparations (to create the 
subnets that this test now expects to be present). I'll merge these changes 
tomorrow when that is done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29276: AWS system test eks and fargate profile: get subnets as external parameter

2023-01-31 Thread via GitHub



o-nikolas commented on PR #29276:
URL: https://github.com/apache/airflow/pull/29276#issuecomment-1411219548

   > For posterity: generally we would prefer to set up and tear down in the 
test, this is only getting changed because of the race condition it was causing.
   
   Good call out :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29276: AWS system test eks and fargate profile: get subnets as external parameter

2023-01-31 Thread via GitHub



o-nikolas commented on PR #29276:
URL: https://github.com/apache/airflow/pull/29276#issuecomment-1411165141

   Agreed, great change. So beautiful when you can just use the variable 
fetcher to hide the nasty bits of creating/managing resources :smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] danielhoherd commented on issue #29273: Status of testing of Apache Airflow Helm Chart 1.8.0rc1

2023-01-31 Thread via GitHub



danielhoherd commented on issue #29273:
URL: https://github.com/apache/airflow/issues/29273#issuecomment-1411148100

   I have validated https://github.com/apache/airflow/pull/27848 which was easy 
to validate. It was just a version bump.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kolfild26 commented on pull request #28998: write to DB a user who triggered a dag via API endpoint request

2023-01-31 Thread via GitHub



kolfild26 commented on PR #28998:
URL: https://github.com/apache/airflow/pull/28998#issuecomment-140613

   Hi, @ephraimbuddy 
   I added test 
[_check_last_log](https://github.com/apache/airflow/pull/28998/commits/e2b670c4382b9e00adece76d5574268d9eb113f3)
 similarly it's done in 
[test_connection_endpoint.py](https://github.com/apache/airflow/blob/5d84b59554c93fd22e92b46a1061b40b899a8dec/tests/api_connexion/endpoints/test_connection_endpoint.py#L84).
   Tests were passed successfully.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



eladkal commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092504689


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -654,8 +673,8 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
 the operation does not timeout.
 :param check_if_job_exists: If set to true, then the operator will check 
whether a training job
 already exists for the name in the config.
-:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "increment"
-(default) and "fail".
+:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "random"
+(default), "increment" (deprecated) and "fail".

Review Comment:
   No, my comment is that we consider value deprecated yet the code does not 
raise deprecation warning.
   If something is deprecated it must have deprecation warning that lead users 
to set the proper values/parameters.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vincbeck opened a new pull request, #29276: AWS system test eks and fargate profile: get subnets as external parameter

2023-01-31 Thread via GitHub



vincbeck opened a new pull request, #29276:
URL: https://github.com/apache/airflow/pull/29276

   In both system tests `example_eks_with_fargate_profile` and 
`example_eks_with_fargate_in_one_step` subnets are created within the system 
test. In order to create these subnets, an available CIDR block is created with 
the function `_get_next_available_cidr`. When both tests are running at the 
same time, some race condition can happen and both tests might pick the same 
CIDR blocks which then result as a failure.
   
   To avoid that, I rather create these subnets outside of the system test and 
fetch them as external parameter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Gollum999 opened a new issue, #29275: Allow user DAGs to emit DagWarning (or similar)

2023-01-31 Thread via GitHub



Gollum999 opened a new issue, #29275:
URL: https://github.com/apache/airflow/issues/29275

   ### Description
   
   I would like a way to emit a warning or error message from a DAG that shows 
up as a banner in the web UI, similar to how exceptions are handled during DAG 
parsing or how 
[`DagWarning`](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dagwarning/index.html)s
 are displayed.
   
   ### Use case/motivation
   
   I have a script under my `dags_folder` that is dynamically discovering and 
building DAGs from a networked filesystem.  If one dynamically discovered DAG 
raises an exception, I don't want that to affect any of the other DAGs that are 
being discovered, so I swallow exceptions and log about them.  However, I have 
not found a good way to communicate to my users that that exception occurred.  
Ideally I think it should look similar to when an exception is raised in a 
traditional DAG - an error banner on the web with (optionally) the full 
traceback.
   
   In my testing, I've found that if *any* exception is raised during DAG 
parsing, it causes *all* DAGs originating from that file to be removed.  So 
this sort of thing does not work:
   ```
   #!/usr/bin/env python3
   from datetime import datetime
   import traceback
   
   from airflow.decorators import dag, task
   
   
   exceptions = []
   for dag_id in 'ABCDE':
   try:
   @dag(
   dag_id=dag_id,
   schedule=None,
   start_date=datetime(2023, 1, 31),
   )
   def test_multi_exception():
   if dag_id in 'BE':
   raise RuntimeError(f'Failed to create dynamic DAG {dag_id}')
   
   @task
   def t():
   print('inside task')
   
   t()
   
   test_multi_exception()
   
   except Exception as e:
   exceptions.append(''.join(traceback.format_exception(etype=type(e), 
value=e, tb=e.__traceback__)))
   
   if exceptions:
   raise RuntimeError('\n'.join(exceptions))
   ```
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a diff in pull request #27758: Enable trigger logging in webserver

2023-01-31 Thread via GitHub



dstandish commented on code in PR #27758:
URL: https://github.com/apache/airflow/pull/27758#discussion_r1092489357


##
airflow/executors/base_executor.py:
##
@@ -355,14 +355,15 @@ def execute_async(
 """
 raise NotImplementedError()
 
-def get_task_log(self, ti: TaskInstance, log: str = "") -> None | str | 
tuple[str, dict[str, bool]]:
+def get_task_log(self, ti: TaskInstance) -> tuple[list[str], list[str]]:

Review Comment:
   this bit
   ```
   response = executor.get_task_log(ti=ti)
   if response:
   executor_messages, executor_logs = response
   ```
   
   it's a remnant from when if `get_task_log` not implemented then would return 
None, but i have since updated baseexecutor implementation to return `[], []` 
so that bit of defensiveness is not strictly necessary, though it doesn't hurt.
   
   i have fixed tests, hopefully not too many more iterations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on pull request #28970: Fix support for macros with dots in DataProcJobBuilder

2023-01-31 Thread via GitHub



Taragolis commented on PR #28970:
URL: https://github.com/apache/airflow/pull/28970#issuecomment-1411061835

   I tried understand initial issue 
https://github.com/apache/airflow/issues/28810 
   
   >> I am passing string containing macros to DataProcJobBuilder task_id 
parameter:
   > {{ (dag.dag_id + '-' + task.task_id.replace('_', '-'))[:90] }}
   
   Is this PR open for allow something like this?
   
   
   ```python
   
   DataProcJobBuilder(
   task_id="{{ (dag.dag_id + '-' + task.task_id.replace('_', '-'))[:90] }}"
   ...
   )
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Github run id:4057758742

2023-01-31 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new fcc02204a4 Updating constraints. Github run id:4057758742
fcc02204a4 is described below

commit fcc02204a42be2d226390c2d16b805dc3dbf6088
Author: Automated GitHub Actions commit 
AuthorDate: Tue Jan 31 20:56:30 2023 +

Updating constraints. Github run id:4057758742

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
'refs/heads/main' in the 'apache/airflow' repository with commit sha 
42fbcb2c10af6a8e7d02e90fdc8e27732dcc04ab.

The action that build those constraints can be found at 
https://github.com/apache/airflow/actions/runs/4057758742/

The image tag used for that build was: 
42fbcb2c10af6a8e7d02e90fdc8e27732dcc04ab. You can enter Breeze environment
with this image by running 'breeze shell --image-tag 
42fbcb2c10af6a8e7d02e90fdc8e27732dcc04ab'

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 20 +++-
 constraints-3.7.txt   | 18 ++
 constraints-3.8.txt   | 20 +++-
 constraints-3.9.txt   | 20 +++-
 constraints-no-providers-3.10.txt |  4 +++-
 constraints-no-providers-3.7.txt  |  4 +++-
 constraints-no-providers-3.8.txt  |  4 +++-
 constraints-no-providers-3.9.txt  |  4 +++-
 constraints-source-providers-3.10.txt | 18 ++
 constraints-source-providers-3.7.txt  | 16 +---
 constraints-source-providers-3.8.txt  | 18 ++
 constraints-source-providers-3.9.txt  | 18 ++
 12 files changed, 94 insertions(+), 70 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 7939895c92..44bd02a830 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2023-01-30T17:34:43Z
+# This constraints file was automatically generated on 2023-01-31T20:55:52Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -171,15 +171,15 @@ azure-synapse-spark==0.7.0
 backcall==0.2.0
 backoff==1.10.0
 bcrypt==4.0.1
-beautifulsoup4==4.11.1
+beautifulsoup4==4.11.2
 billiard==3.6.4.0
 bitarray==2.6.2
 black==23.1a1
 bleach==6.0.0
 blinker==1.5
-boto3==1.26.59
+boto3==1.26.61
 boto==2.49.0
-botocore==1.29.59
+botocore==1.29.61
 bowler==0.9.0
 cachelib==0.9.0
 cachetools==4.2.2
@@ -272,6 +272,7 @@ google-cloud-compute==0.7.0
 google-cloud-container==2.11.1
 google-cloud-core==2.3.2
 google-cloud-datacatalog==3.9.0
+google-cloud-dataflow-client==0.5.4
 google-cloud-dataform==0.2.0
 google-cloud-dataplex==1.1.0
 google-cloud-dataproc-metastore==1.6.0
@@ -317,7 +318,7 @@ httplib2==0.20.4
 httpx==0.23.3
 humanize==4.5.0
 hvac==1.0.2
-identify==2.5.16
+identify==2.5.17
 idna==3.4
 ijson==3.2.0.post0
 imagesize==1.4.1
@@ -351,6 +352,7 @@ jupyter_core==5.2.0
 keyring==23.13.1
 kombu==5.2.4
 krb5==0.4.1
+kubernetes-asyncio==24.2.2
 kubernetes==23.6.0
 kylinpy==2.8.4
 lazy-object-proxy==1.9.0
@@ -375,7 +377,7 @@ more-itertools==9.0.0
 moreorless==0.4.0
 moto==4.1.1
 msal-extensions==1.0.0
-msal==1.20.0
+msal==1.21.0
 msgpack==1.0.4
 msrest==0.7.1
 msrestazure==0.6.4
@@ -510,7 +512,7 @@ rfc3986==1.5.0
 rich-click==1.6.1
 rich==13.3.1
 rsa==4.9
-ruff==0.0.237
+ruff==0.0.238
 s3transfer==0.6.0
 sarif-om==1.0.4
 sasl==0.3.1
@@ -541,7 +543,7 @@ sphinx-jinja==2.0.2
 sphinx-rtd-theme==1.1.1
 sphinxcontrib-applehelp==1.0.4
 sphinxcontrib-devhelp==1.0.2
-sphinxcontrib-htmlhelp==2.0.0
+sphinxcontrib-htmlhelp==2.0.1
 sphinxcontrib-httpdomain==1.8.1
 sphinxcontrib-jsmath==1.0.1
 sphinxcontrib-qthelp==1.0.3
@@ -549,7 +551,7 @@ sphinxcontrib-redoc==1.6.0
 sphinxcontrib-serializinghtml==1.1.5
 sphinxcontrib-spelling==7.7.0
 spython==0.3.0
-sqlalchemy-bigquery==1.5.0
+sqlalchemy-bigquery==1.6.0
 sqlalchemy-drill==1.1.2
 sqlalchemy-redshift==0.8.12
 sqlparse==0.4.3
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 55a5319494..cffeee1bf4 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2023-01-30T17:35:15Z
+# This constraints file was automatically generated on 2023-01-31T20:56:27Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'

[GitHub] [airflow] shahar1 commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator

2023-01-31 Thread via GitHub



shahar1 commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-1411052265

   @eladkal Please assign me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on pull request #28869: rewrite polling code for appflow hook

2023-01-31 Thread via GitHub



Taragolis commented on PR #28869:
URL: https://github.com/apache/airflow/pull/28869#issuecomment-1411051606

   @o-nikolas Oh, I really do not use Appflow. I've just ping Igor, because 
remember (and git blame told me) that he initially implement. But let me have a 
look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #29243: Auto refresh DAG state in UI

2023-01-31 Thread via GitHub



potiuk commented on issue #29243:
URL: https://github.com/apache/airflow/issues/29243#issuecomment-1411044008

   I see. Makes sense. Thanks @bbovenzi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Gollum999 commented on issue #27399: CronTriggerTimetable lost one task occasionally

2023-01-31 Thread via GitHub



Gollum999 commented on issue #27399:
URL: https://github.com/apache/airflow/issues/27399#issuecomment-1411036858

   For what it's worth, prior to 2.4 I wrote my own timetable that acts almost 
exactly like `CronTriggerTimetable`, and I experienced a similar issue, so I'll 
mention my findings here.
   
   In my case the problem was that there was a race condition between the 
DagProcessor and the Scheduler when calling `Timetable.next_dagrun_info`. 
Depending on when each thread evaluated the timetable, the DagProcessor could 
end up "bumping" the DAG's `next_dagrun_create_after` timestamp in the DB 
before the Scheduler had a chance to schedule the current interval.  Then when 
the Scheduler would do it's query to determine which DagRuns to schedule, it 
wouldn't find the run that got "skipped".  In my case, I believe the bug was a 
call to `DateTime.utcnow()` that I had to align to the previous interval 
boundary.
   
   Also worth noting that my bug only happened with `catchup=False`, but that 
could have just been a quirk with my implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29143: Demonstrate usage of the PythonSensor

2023-01-31 Thread via GitHub



potiuk commented on PR #29143:
URL: https://github.com/apache/airflow/pull/29143#issuecomment-1411034438

   Exactly same effect, example is runnable with `--load-example-dags` flag in 
start-airflow. No code duplication.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #29143: Demonstrate usage of the PythonSensor

2023-01-31 Thread via GitHub



potiuk commented on PR #29143:
URL: https://github.com/apache/airflow/pull/29143#issuecomment-1411031392

   > I suggest showing reproducible examples. The current example for the 
PythonSensor is not easily reproducible because it doesn't show imports.
   
   Yes I understand that. 
   
   What I am suggesting to also surround import with "exaampleinclude" start. 
There is no reason exampleicnlude start/end comments shoud be limited to in-dag.
   
   the new `example_python_sensors.dag` could look like this (and gradually we 
could extract other sensors in similar way to separate files):
   
   ```
   # Licensed to the Apache Software Foundation (ASF) under one
   # or more contributor license agreements.  See the NOTICE file
   # distributed with this work for additional information
   # regarding copyright ownership.  The ASF licenses this file
   # to you under the Apache License, Version 2.0 (the
   # "License"); you may not use this file except in compliance
   # with the License.  You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing,
   # software distributed under the License is distributed on an
   # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   # KIND, either express or implied.  See the License for the
   # specific language governing permissions and limitations
   # under the License.
   
   # [START example_python_sensors]
   import datetime
   from airflow.decorators import dag, task
   from airflow.sensors.python import PythonSensor
   @dag(start_date=datetime.datetime(2023, 1, 1), schedule=None)
   def example():
   # TaskFlow sensor
   @task.sensor
   def wait_for_success_taskflow():
   return datetime.datetime.now().minute % 2 == 0
   wait_for_success_taskflow()
   # Equivalent functionality using PythonSensor class
   def wait_for_success_pythonsensor():
   return datetime.datetime.now().minute % 2 == 0
   PythonSensor(task_id="wait_for_success_pythonsensor", 
python_callable=wait_for_success_pythonsensor)
   example()
   # [END example_python_sensors]
   ```
   (whole file).
   
   The weekday
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz opened a new pull request, #29274: log the observed status in redshift sensor

2023-01-31 Thread via GitHub



vandonr-amz opened a new pull request, #29274:
URL: https://github.com/apache/airflow/pull/29274

   this sensor only checks against the "wanted" status, and not other possibly 
terminal statuses.
   So if the cluster takes an unexpected branch and ends up in a semi-terminal 
state, this is completely invisible, and would only fail with an uninformative 
timeout after a long time.
   
   The "proper" fix would be to have a list of terminal states to check 
against, but I don't have enough redshift knowledge to know what's terminal in 
the long list of possible statuses:
   
   - available
   - available, prep-for-resize
   - available, resize-cleanup
   - cancelling-resize
   - creating
   - deleting
   - final-snapshot
   - hardware-failure
   - incompatible-hsm
   - incompatible-network
   - incompatible-parameters
   - incompatible-restore
   - modifying
   - paused
   - rebooting
   - renaming
   - resizing
   - rotating-keys
   - storage-full
   - updating-hsm
   
   So I'm taking this poor man's approach that'd at least allow us to get a 
sense of what happened from looking at the logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham opened a new issue, #29273: Status of testing of Apache Airflow Helm Chart 1.8.0rc1

2023-01-31 Thread via GitHub



jedcunningham opened a new issue, #29273:
URL: https://github.com/apache/airflow/issues/29273

   ### Body
   
   We have a kind request for all the contributors to the latest [Apache 
Airflow Helm Chart 
1.8.0rc1](https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.8.0rc1).
   
   Could you please help us to test the RC versions of the Apache Airflow Helm 
Chart?
   
   Please let us know in the comment if the issue is addressed in the latest RC.
   
   - [ ] [Add nodePort declaration to chart/values.schema.json 
(#26945)](https://github.com/apache/airflow/pull/26945): @joseph-max-coalfire
Linked issues:
- [Add NodePort Option to the values schema 
(#26812)](https://github.com/apache/airflow/issues/26812)
   - [ ] [Make custom env vars optional for job templates 
(#27148)](https://github.com/apache/airflow/pull/27148): @richardbadman @tseruga
Linked issues:
- [[Helm Chart] Decouple custom_airflow_environment from all pods and 
jobs (#26045)](https://github.com/apache/airflow/issues/26045)
   - [ ] [Add workeer log-groomer-sidecar enable option in helm chart 
(#27178)](https://github.com/apache/airflow/pull/27178): @BobDu @jedcunningham
Linked issues:
- [Adds support for custom command and args 
(#16153)](https://github.com/apache/airflow/pull/16153)
   - [ ] [Chart: Default airflow version to 2.4.2 
(#27230)](https://github.com/apache/airflow/pull/27230): @ephraimbuddy
   - [ ] [Add webserverConfigConfigMapName 
(#27419)](https://github.com/apache/airflow/pull/27419): @JulesTriomphe
   - [ ] [ExtraVolumeMounts in sidecars and initContainers 
(#27420)](https://github.com/apache/airflow/pull/27420): @JulesTriomphe
   - [ ] [allow pgbouncer replicas to be configurable 
(#27439)](https://github.com/apache/airflow/pull/27439): @pgvishnuram
   - [ ] [[BUMP] helm chart - redis 7 
(#27443)](https://github.com/apache/airflow/pull/27443): @raphaelauv
   - [ ] [Chart: add hostaliases to pod template file 
(#27544)](https://github.com/apache/airflow/pull/27544): @Aakcht
   - [ ] [Chart: Decrease default gitSync wait 
(#27625)](https://github.com/apache/airflow/pull/27625): @jedcunningham
   - [ ] [Chart: Update default airflow version to 2.4.3 
(#27668)](https://github.com/apache/airflow/pull/27668): @ephraimbuddy
   - [ ] [Correct repo example for cloning via ssh - comments in helm chart 
(#27671)](https://github.com/apache/airflow/pull/27671): @schudoku @morkot
Linked issues:
- [helm chart - 
mounting-dags-from-a-private-github-repo-using-git-sync-sidecar 
(#21970)](https://github.com/apache/airflow/issues/21970)
   - [ ] [Replace helm chart icon 
(#27704)](https://github.com/apache/airflow/pull/27704): @ronahk
   - [ ] [Chart: add airflow_local_settings to all airflow containers 
(#27779)](https://github.com/apache/airflow/pull/27779): @rauljmz @Aakcht 
@venkatnpedada
Linked issues:
- [Not able to use custom logging with airflow helm chart 
(#27498)](https://github.com/apache/airflow/issues/27498)
- [Cannot reference airflow_local_settings in config settings 
(#25637)](https://github.com/apache/airflow/issues/25637)
   - [ ] [Add global volume & volumeMounts to the chart 
(#27781)](https://github.com/apache/airflow/pull/27781): @csp33
Linked issues:
- [Allow global volume & volumeMounts in Helm chart 
(#27687)](https://github.com/apache/airflow/issues/27687)
   - [ ] [git-sync 3.6.3 
(#27848)](https://github.com/apache/airflow/pull/27848): @danielhoherd
   - [ ] [Enhance chart to allow over-riding command-line args to statsd 
exporter (#28041)](https://github.com/apache/airflow/pull/28041): @rob-1126
   - [ ] [Chart: Default airflow version to 2.5.0 
(#28074)](https://github.com/apache/airflow/pull/28074): @ephraimbuddy
   - [ ] [Fix helm chart NOTES.txt to show correct URL 
(#28264)](https://github.com/apache/airflow/pull/28264): @okue
   - [ ] [Remove checks for 1.19 api checks 
(#28461)](https://github.com/apache/airflow/pull/28461): @potiuk @Jmdaming
Linked issues:
- [Wrong apiVersion Detected During Ingress Creation 
(#22657)](https://github.com/apache/airflow/issues/22657)
   - [ ] [Add missing annotations on deployments [#28604] 
(#28688)](https://github.com/apache/airflow/pull/28688): @sudeepgupta90
Linked issues:
- [Missing annotations on Deployments 
(#28604)](https://github.com/apache/airflow/issues/28604)
   - [ ] [Add worker service account for LocalKubernetesExecutor 
(#28813)](https://github.com/apache/airflow/pull/28813): @Chen-Oliver
   - [ ] [Fix helm chart create scheduler service to  serve task logs for 
LocalKubernetesExecutor 
(#28828)](https://github.com/apache/airflow/pull/28828): @snjypl
   - [ ] [Fix incorrect comments in worker-kedaautoscaler.yaml 
(#28858)](https://github.com/apache/airflow/pull/28858): @EricGao888
   - [ ] [Chart: add doc note about podtemplate images 
(#29032)](https://github.com/apache/airflow/pull/29032): @Aakcht
   - [ ]

[airflow] annotated tag helm-chart/1.8.0rc1 updated (bca7816f41 -> b17bd670ca)

2023-01-31 Thread jedcunningham

This is an automated email from the ASF dual-hosted git repository.

jedcunningham pushed a change to annotated tag helm-chart/1.8.0rc1
in repository https://gitbox.apache.org/repos/asf/airflow.git


*** WARNING: tag helm-chart/1.8.0rc1 was modified! ***

from bca7816f41 (commit)
  to b17bd670ca (tag)
 tagging bca7816f41a190398ecb2e76f9e53380c61052ac (commit)
 replaces providers-apache-cassandra/3.1.1
  by Jed Cunningham
  on Tue Jan 31 12:51:48 2023 -0700

- Log -
Apache Airflow Helm Chart 1.8.0rc1
-BEGIN PGP SIGNATURE-

iQIzBAABCAAdFiEE4aHphPVbjygL2cuiC7cWOJKi5I4FAmPZcVQACgkQC7cWOJKi
5I4SEA/+PIWxSmTO/ib0vUaVhj1jXormTySNcK32KZPyRbNLYD/RO/3YR5n7/Ih/
ANdtA0XT6AMwxoBcOTNsMmlRgIoo1esGcnx24vIUhe1T4caF9C3oB2Ew+wt8hf3A
gMCklewEsloa+wd1DE54YGfvWy7sFfa/2PglUmS3h/zOta0fjQdda9Jrf6hk8zMT
dlcQ9oWNMK4rv2VfVfY+lEO8dkCqSnL4wTnviP5fK4l1tWruyRdRH/WCMisswfiK
6n784viR1bnxyiWxNP6b1fFD4FvVURmdMnfiySmaR+fTMl6Zwpy/5tUuRQAyvZCL
Ej1Zz7deMgc74xSZRE4KpKbewDDWNCUDeMsulQpaEAGhg9xkXMmxPAZIgom8aQD0
77JBIw+bigmsZkVqBKG8fHpb90OoVv2qXFa9vdX8tc8XWik/UKVU7zSERI3czpb5
4+97yijHKmqcb0/NdbAEBphLRgxfJRkGIrCC7ZeiPcbcdISv5laQzJHzyciouEO+
ExcTV2pgzzPUeDi7ejUM7KoGhFRyF45UipTYe/HGcmAdmdd3IVfUfUgtBKYZQLFi
0ualkbmTGafBlihoxp9aNcgjnGEQ/xyGYe327w6vtEcbn6Vs1WsGC2SP/Nvndm5i
TMEJ0H/qyQnaP0Xl/UhLcpIzlHHao+4iPO5Oo/TQ9rr4S5I+OGw=
=Pk2L
-END PGP SIGNATURE-
---


No new revisions were added by this update.

Summary of changes:

svn commit: r59767 - /dev/airflow/helm-chart/1.7.0rc1/

2023-01-31 Thread jedcunningham

Author: jedcunningham
Date: Tue Jan 31 20:02:44 2023
New Revision: 59767

Log:
Remove old Helm Chart release: 1.7.0rc1

Removed:
dev/airflow/helm-chart/1.7.0rc1/

svn commit: r59766 - /dev/airflow/helm-chart/1.8.0rc1/

2023-01-31 Thread jedcunningham

Author: jedcunningham
Date: Tue Jan 31 20:01:47 2023
New Revision: 59766

Log:
Add artifacts for Helm Chart 1.8.0rc1

Added:
dev/airflow/helm-chart/1.8.0rc1/
dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz   (with props)
dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.asc
dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.prov
dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.sha512
dev/airflow/helm-chart/1.8.0rc1/airflow-chart-1.8.0-source.tar.gz   (with 
props)
dev/airflow/helm-chart/1.8.0rc1/airflow-chart-1.8.0-source.tar.gz.asc
dev/airflow/helm-chart/1.8.0rc1/airflow-chart-1.8.0-source.tar.gz.sha512
dev/airflow/helm-chart/1.8.0rc1/index.yaml

Added: dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz
==
Binary file - no diff available.

Propchange: dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz
--
svn:mime-type = application/octet-stream

Added: dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.asc
==
--- dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.asc (added)
+++ dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.asc Tue Jan 31 20:01:47 
2023
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+
+iQJNBAABCAA3FiEE4aHphPVbjygL2cuiC7cWOJKi5I4FAmPZcdMZHGplZGN1bm5p
+bmdoYW1AYXBhY2hlLm9yZwAKCRALtxY4kqLkjngXD/47zRVyaslXs215RKalqgDf
+X/frayKP7V0XO57CMpJK8T7lh05ncZuwH26QmtHQxJ0BYR46v25Mqk4kwaMXUGWN
+fKEb+RozwECaTYaiY9YYXgP3iU9ojaPYomwb942zhkphOSK4G/z9AFqSAsUq4l8r
+W/FKEt3fb13daxFojTeuv70zq6p0sGekxasO4Hr/43dLAawM7urf3A2SuMsgx5ik
+wwX/nvS5xa48Hib0pUDjGCbwVKcvPqHNzit2NKGViskbxBNMU/jagS7g5GoRaw+q
+Wtm3DV3YJAz3lrOoEneiaZP8uzlzsP4w1B4SB2fN/LxXPgjWyK/DmgG30RLKXAfT
+ZOPcL82Arh3iy9BYyOXXPsIIeACcAOHPji3ap6ecz+C6SkZuXNFu9zYrpvSnWEpF
+Dlv9dicLvRINeJdIjsRxL52Tfsne9JWIb6+o1jbQ1FLkJiBmaLJJmH5ueV2kbiui
+Na2uxQuBHloi9JL/991HECiXz469J0wmpk3dnbihjEaRCMBGoox2HIv0bOwidXlX
+dSoHKLTFgzaLbxdM9tg65irRia5GkAXKZkI9AwRou/sjJb8a4bbSlEpMezznqpwm
+nEsxQqHwQ8y/m28Rgga3qGwlhre4cHPnRSzGhnMU2lbbjkB8e0Pua4Tr2qAlr7kP
+q4mE7nXyGrHYNmhnZRoGPA==
+=sw4a
+-END PGP SIGNATURE-

Added: dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.prov
==
--- dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.prov (added)
+++ dev/airflow/helm-chart/1.8.0rc1/airflow-1.8.0.tgz.prov Tue Jan 31 20:01:47 
2023
@@ -0,0 +1,214 @@
+-BEGIN PGP SIGNED MESSAGE-
+Hash: SHA256
+
+annotations:
+  artifacthub.io/changes: |
+- description: Add annotations on deployments
+  kind: added
+  links:
+  - name: '#28688'
+url: https://github.com/apache/airflow/pull/28688
+- description: Add global volume & volumeMounts to the chart
+  kind: added
+  links:
+  - name: '#27781'
+url: https://github.com/apache/airflow/pull/27781
+- description: Add support for ``webserverConfigConfigMapName``
+  kind: changed
+  links:
+  - name: '#27419'
+url: https://github.com/apache/airflow/pull/27419
+- description: Enhance chart to allow overriding command-line args to 
statsd exporter
+  kind: changed
+  links:
+  - name: '#28041'
+url: https://github.com/apache/airflow/pull/28041
+- description: Add support for NodePort in Services
+  kind: changed
+  links:
+  - name: '#26945'
+url: https://github.com/apache/airflow/pull/26945
+- description: Add worker log-groomer-sidecar enable option
+  kind: changed
+  links:
+  - name: '#27178'
+url: https://github.com/apache/airflow/pull/27178
+- description: Add HostAliases to Pod template file
+  kind: changed
+  links:
+  - name: '#27544'
+url: https://github.com/apache/airflow/pull/27544
+- description: Allow PgBouncer replicas to be configurable
+  kind: changed
+  links:
+  - name: '#27439'
+url: https://github.com/apache/airflow/pull/27439
+- description: Create scheduler service to serve task logs for 
LocalKubernetesExecutor
+  kind: fixed
+  links:
+  - name: '#28828'
+url: https://github.com/apache/airflow/pull/28828
+- description: Fix NOTES.txt to show correct URL
+  kind: fixed
+  links:
+  - name: '#28264'
+url: https://github.com/apache/airflow/pull/28264
+- description: Add worker service account for LocalKubernetesExecutor
+  kind: fixed
+  links:
+  - name: '#28813'
+url: https://github.com/apache/airflow/pull/28813
+- description: Remove checks for 1.19 api checks
+  kind: fixed
+  links:
+  - name: '#28461'
+url: https://github.com/apache/airflow/pull/28461
+- description: Add airflow_local_settings to all airflow containers
+  kind: fixed
+  links:
+  - name: '#27779'
+url: ht

[GitHub] [airflow] ferruzzi commented on pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



ferruzzi commented on PR #29245:
URL: https://github.com/apache/airflow/pull/29245#issuecomment-1410974971

   I guess I see what you are saying.  With a reasonably large range on the 
random suffix the collision rate will be low.  My initial thought was how many 
times our proposed name might collide with a random suffix, but that's not 
really a big deal.  I think I still like the incremental suffix, but I can see 
your point... I won't stand in the way if others think the tradeoff is worth 
it. :+1: 
   
   Also, you've been doing a lot of little efficiency tweaks; it's definitely 
nice to see them. :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (b3536aeac9 -> 42fbcb2c10)

2023-01-31 Thread jedcunningham

This is an automated email from the ASF dual-hosted git repository.

jedcunningham pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from b3536aeac9 Release notes for helm chart 1.8.0 (#29248)
 add 42fbcb2c10 Fix helm chart release notes code block (#29271)

No new revisions were added by this update.

Summary of changes:
 chart/RELEASE_NOTES.rst | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

[GitHub] [airflow] jedcunningham merged pull request #29271: Fix helm chart release notes code block

2023-01-31 Thread via GitHub



jedcunningham merged PR #29271:
URL: https://github.com/apache/airflow/pull/29271


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] taragurung opened a new issue, #29272: Airflow Dag status failed with no logs

2023-01-31 Thread via GitHub



taragurung opened a new issue, #29272:
URL: https://github.com/apache/airflow/issues/29272

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   All of a sudden the DAG started failing and the problem is there are no 
informative logs to fix the issues as following:
   ```
   *** Log file does not exist: 
/opt/airflow/logs/TEST_V2/startNoteBookRun/2023-01-31T19:28:16.029295+00:00/1.log
   *** Fetching from: 
http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/TEST_V2/startNoteBookRun/2023-01-31T19:28:16.029295+00:00/1.log
   
   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1043} INFO - Dependencies all 
met for 
   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1043} INFO - Dependencies all 
met for 
   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1249} INFO - 
   

   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1250} INFO - Starting attempt 1 
of 2
   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1251} INFO - 
   

   [2023-01-31, 19:28:24 UTC] {taskinstance.py:1270} INFO - Executing 
 on 2023-01-31 19:28:16.029295+00:00
   [2023-01-31, 19:28:24 UTC] {standard_task_runner.py:52} INFO - Started 
process 44 to run task
   [2023-01-31, 19:28:24 UTC] {standard_task_runner.py:79} INFO - Running: 
['airflow', 'tasks', 'run', 'TEST_V2', 'startNoteBookRun', 
'manual__2023-01-31T19:28:16.029295+00:00', '--job-id', '93', '--raw', 
'--subdir', 'DAGS_FOLDER/TEST_v2.py', '--cfg-path', '/tmp/tmpyd5hi8cd', 
'--error-file', '/tmp/tmp7l8fcqa6']
   [2023-01-31, 19:28:24 UTC] {standard_task_runner.py:80} INFO - Job 93: 
Subtask startNoteBookRun
   [2023-01-31, 19:28:25 UTC] {logging_mixin.py:109} INFO - Running 
 on host 
airflow-worker-0.airflow-worker.airflow.svc.cluster.local
   [2023-01-31, 19:28:25 UTC] {taskinstance.py:1448} INFO - Exporting the 
following env vars:
   AIRFLOW_CTX_DAG_EMAIL=t...@xyz.com
   AIRFLOW_CTX_DAG_OWNER=airflow
   AIRFLOW_CTX_DAG_ID=APP_POZE_TEREN_V2
   AIRFLOW_CTX_TASK_ID=startNoteBookRun
   AIRFLOW_CTX_EXECUTION_DATE=2023-01-31T19:28:16.029295+00:00
   AIRFLOW_CTX_DAG_RUN_ID=manual__2023-01-31T19:28:16.029295+00:00
   [2023-01-31, 19:28:25 UTC] {logging_mixin.py:109} INFO - Env: DEV
   [2023-01-31, 19:28:25 UTC] {logging_mixin.py:109} INFO - STARTING VERTEX 
CUSTOM JOB...
   ```
   
   
   
   ### What you think should happen instead
   
   Dags running successfully as all service pods are running with no issues. 
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
2.2.5
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Kubernetes
   
   ### Anything else
   
   1. Tried changing the logging to ERRO and DEBUG as well but nothing showed 
up to figure out the cause of the problems
   2. All the pods of scheduler / workers / and all are running fine
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29272: Airflow Dag status failed with no logs

2023-01-31 Thread boring-cyborg



boring-cyborg[bot] commented on issue #29272:
URL: https://github.com/apache/airflow/issues/29272#issuecomment-1410966021

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



vandonr-amz commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092398234


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -654,8 +673,8 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
 the operation does not timeout.
 :param check_if_job_exists: If set to true, then the operator will check 
whether a training job
 already exists for the name in the config.
-:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "increment"
-(default) and "fail".
+:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "random"
+(default), "increment" (deprecated) and "fail".

Review Comment:
   so, I don't think that's the direction this PR is taking, but if I were to 
keep the code as-is, changing this to `"increment" (deprecated, use "random" 
instead)` would be good for you ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



vandonr-amz commented on PR #29245:
URL: https://github.com/apache/airflow/pull/29245#issuecomment-1410954197

   > We're only counting jobs whose name matches 
[here](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/sagemaker.py#L1022)
   
   Oh yes my bad, I read the code too fast.
   
   Having a random number rather than beautiful sequence is not better in 
itself of course, but I think the price to pay for this little nice thing is 
too high.
   
   Instead of an O(1) operation, we end up doing potentially hundreds of 
network calls, with added wait time to account for throttling errors, every 
time we create a job. All this just so that we can eventually have a nice 
number in a list that is ordered by creation date by default anyway.
   We also run the risk of collisions if two jobs with the same name are 
created at the same time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



eladkal commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092387832


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -654,8 +673,8 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
 the operation does not timeout.
 :param check_if_job_exists: If set to true, then the operator will check 
whether a training job
 already exists for the name in the config.
-:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "increment"
-(default) and "fail".
+:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "random"
+(default), "increment" (deprecated) and "fail".

Review Comment:
   My concern is specifically about pointing the (deprecated)
   usually if we deprecate something, while immediate action isn't required 
because we are backward compatible an action is required to be done for the 
next major release (when we are making breaking change). so if we say something 
is deprecated we must also say what to do, otherwise this is just inviting many 
questions from users who are not sure what is expected from them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



vandonr-amz commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092382659


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -654,8 +673,8 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
 the operation does not timeout.
 :param check_if_job_exists: If set to true, then the operator will check 
whether a training job
 already exists for the name in the config.
-:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "increment"
-(default) and "fail".
+:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "random"
+(default), "increment" (deprecated) and "fail".

Review Comment:
   I didn't because the behavior is the same as "random" in my proposed code, 
so there is really no specific action required from users, I'd just prefer new 
code to use the "right" value.
   But if we want to maintain the old behavior, then I'd add a warning



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



ferruzzi commented on PR #29245:
URL: https://github.com/apache/airflow/pull/29245#issuecomment-1410943198

   I don't think I agree that using a random suffix is an improvement.
   
   >  The sequence was already imperfect, because we counted all jobs, not just 
the ones with the same base name, so it'd go like:
   
   We're only counting jobs whose name matches 
[here](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/sagemaker.py#L1022)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



eladkal commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092374357


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -654,8 +673,8 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
 the operation does not timeout.
 :param check_if_job_exists: If set to true, then the operator will check 
whether a training job
 already exists for the name in the config.
-:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "increment"
-(default) and "fail".
+:param action_if_job_exists: Behaviour if the job name already exists. 
Possible options are "random"
+(default), "increment" (deprecated) and "fail".

Review Comment:
   I may have missed it. Where did we raise deprecation warning for this value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



ferruzzi commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092368310


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -679,11 +697,11 @@ def __init__(
 self.check_interval = check_interval
 self.max_ingestion_time = max_ingestion_time
 self.check_if_job_exists = check_if_job_exists
-if action_if_job_exists in ("increment", "fail"):
+if action_if_job_exists in {"random", "increment", "fail"}:

Review Comment:
   This was my recent poke at this code from back in November:  
https://github.com/apache/airflow/pull/27634



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



vandonr-amz commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092360487


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -679,11 +697,11 @@ def __init__(
 self.check_interval = check_interval
 self.max_ingestion_time = max_ingestion_time
 self.check_if_job_exists = check_if_job_exists
-if action_if_job_exists in ("increment", "fail"):
+if action_if_job_exists in {"random", "increment", "fail"}:

Review Comment:
   also, the "shape" of the naming is still the same, if a regex had been 
designed for the old code, the new code would pass it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



vandonr-amz commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092359626


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -679,11 +697,11 @@ def __init__(
 self.check_interval = check_interval
 self.max_ingestion_time = max_ingestion_time
 self.check_if_job_exists = check_if_job_exists
-if action_if_job_exists in ("increment", "fail"):
+if action_if_job_exists in {"random", "increment", "fail"}:

Review Comment:
   well there is the "every change is a breaking change" philosophy of course, 
and _maybe_ I can imagine a workflow where changing this would be breaking, but 
it'd be pretty far fetched.
   I can always keep the existing behavior, but I'd put big deprecation 
warnings around it, because let's say you run 10 jobs a day, you'd be looking 
at throttling exceptions after 2-3 years, which is not such a long time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #29144: rewrite invoke lamba operator tests

2023-01-31 Thread via GitHub



o-nikolas commented on PR #29144:
URL: https://github.com/apache/airflow/pull/29144#issuecomment-1410906782

   Hey folks,
   
   What's the status on this one? @Taragolis are you happy to approve the PR as 
it is now, or would you like to see some changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-01-31 Thread via GitHub



o-nikolas commented on code in PR #29245:
URL: https://github.com/apache/airflow/pull/29245#discussion_r1092353835


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -679,11 +697,11 @@ def __init__(
 self.check_interval = check_interval
 self.max_ingestion_time = max_ingestion_time
 self.check_if_job_exists = check_if_job_exists
-if action_if_job_exists in ("increment", "fail"):
+if action_if_job_exists in {"random", "increment", "fail"}:

Review Comment:
   These will cause the jobs to have different names from now onward, ya? I 
think @ferruzzi modified this code somewhat recently and we tried to keep the 
naming the same for backwards compatibility (folks may have other logic that 
depends on the names being in the format they have been historically).
   How to handle these low level breaking changes has been a hot topic lately 
in Airflow, I still don't think we've landed on a spot everyone agrees with. 
Thoughts?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Release notes for helm chart 1.8.0 (#29248)

2023-01-31 Thread jedcunningham

This is an automated email from the ASF dual-hosted git repository.

jedcunningham pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new b3536aeac9 Release notes for helm chart 1.8.0 (#29248)
b3536aeac9 is described below

commit b3536aeac9823c2d3b9c807b5ce01a06252fb4b3
Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com>
AuthorDate: Tue Jan 31 12:55:21 2023 -0600

Release notes for helm chart 1.8.0 (#29248)
---
 chart/Chart.yaml  | 226 --
 chart/RELEASE_NOTES.rst   | 105 +-
 chart/newsfragments/27353.significant.rst |  21 ---
 chart/newsfragments/27625.significant.rst |   4 -
 chart/newsfragments/29074.significant.rst |   3 -
 newsfragments/27698.significant.rst   |   1 -
 6 files changed, 194 insertions(+), 166 deletions(-)

diff --git a/chart/Chart.yaml b/chart/Chart.yaml
index 028837cfa9..ba5c27f09b 100644
--- a/chart/Chart.yaml
+++ b/chart/Chart.yaml
@@ -44,7 +44,7 @@ type: application
 annotations:
   artifacthub.io/links: |
 - name: Documentation
-  url: https://airflow.apache.org/docs/helm-chart/1.6.0/
+  url: https://airflow.apache.org/docs/helm-chart/1.8.0/
   artifacthub.io/screenshots: |
 - title: DAGs View
   url: 
https://airflow.apache.org/docs/apache-airflow/2.5.1/_images/dags.png
@@ -67,187 +67,143 @@ annotations:
 - title: Task Instance Context Menu
   url: 
https://airflow.apache.org/docs/apache-airflow/2.5.1/_images/context.png
   artifacthub.io/changes: |
-- description: Make cleanup job history configurable
+- description: Add annotations on deployments
   kind: added
   links:
-  - name: '#26838'
-url: https://github.com/apache/airflow/pull/26838
-- description: Added labels to specific Airflow components
+  - name: '#28688'
+url: https://github.com/apache/airflow/pull/28688
+- description: Add global volume & volumeMounts to the chart
   kind: added
   links:
-  - name: '#25031'
-url: https://github.com/apache/airflow/pull/25031
-- description: Add StatsD ``overrideMappings`` in Helm chart values
-  kind: added
-  links:
-  - name: '#26598'
-url: https://github.com/apache/airflow/pull/26598
-- description: Adding ``podAnnotations`` to StatsD deployment template
-  kind: added
-  links:
-  - name: '#25732'
-url: https://github.com/apache/airflow/pull/25732
-- description: Container specific extra environment variables
-  kind: added
-  links:
-  - name: '#24784'
-url: https://github.com/apache/airflow/pull/24784
-- description: Custom labels for extrasecrets/configmaps
-  kind: added
-  links:
-  - name: '#25283'
-url: https://github.com/apache/airflow/pull/25283
-- description: Add ``revisionHistoryLimit`` to all deployments
-  kind: added
-  links:
-  - name: '#25059'
-url: https://github.com/apache/airflow/pull/25059
-- description: Adding ``podAnnotations`` to Redis StatefulSet
-  kind: added
-  links:
-  - name: '#23708'
-url: https://github.com/apache/airflow/pull/23708
-- description: Provision Standalone Dag Processor
-  kind: added
-  links:
-  - name: '#23711'
-url: https://github.com/apache/airflow/pull/23711
-- description: Add configurable scheme for webserver probes
-  kind: added
+  - name: '#27781'
+url: https://github.com/apache/airflow/pull/27781
+- description: Add support for ``webserverConfigConfigMapName``
+  kind: changed
   links:
-  - name: '#22815'
-url: https://github.com/apache/airflow/pull/22815
-- description: Add support for KEDA HPA config to Helm chart
-  kind: added
+  - name: '#27419'
+url: https://github.com/apache/airflow/pull/27419
+- description: Enhance chart to allow overriding command-line args to 
statsd exporter
+  kind: changed
   links:
-  - name: '#24220'
-url: https://github.com/apache/airflow/pull/24220
-- description: Add 'executor' label to Airflow scheduler deployment
+  - name: '#28041'
+url: https://github.com/apache/airflow/pull/28041
+- description: Add support for NodePort in Services
   kind: changed
   links:
-  - name: '#25684'
-url: https://github.com/apache/airflow/pull/25684
-- description: Add default ``flower_url_prefix`` in Helm chart values
+  - name: '#26945'
+url: https://github.com/apache/airflow/pull/26945
+- description: Add worker log-groomer-sidecar enable option
   kind: changed
   links:
-  - name: '#26415'
-url: https://github.com/apache/airflow/pull/26415
-- description: Add liveness probe to Celery workers
+  - name: '#27178'
+url: https://githu

[GitHub] [airflow] jedcunningham merged pull request #29248: Release notes for helm chart 1.8.0

2023-01-31 Thread via GitHub



jedcunningham merged PR #29248:
URL: https://github.com/apache/airflow/pull/29248


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #29202: Trigger rule one_success with dynamic task mapping

2023-01-31 Thread via GitHub



eladkal commented on issue #29202:
URL: https://github.com/apache/airflow/issues/29202#issuecomment-1410900752

   I agree with @hussein-awala explnation thus Im closing this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal closed issue #29202: Trigger rule one_success with dynamic task mapping

2023-01-31 Thread via GitHub



eladkal closed issue #29202: Trigger rule one_success with dynamic task mapping
URL: https://github.com/apache/airflow/issues/29202


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on pull request #28869: rewrite polling code for appflow hook

2023-01-31 Thread via GitHub



o-nikolas commented on PR #28869:
URL: https://github.com/apache/airflow/pull/28869#issuecomment-1410894943

   @Taragolis @igorborgest are you happy with the state of this PR or would you 
like to see any other changes/testing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (42dd812021 -> a671561b8e)

2023-01-31 Thread onikolas

This is an automated email from the ASF dual-hosted git repository.

onikolas pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 42dd812021 Add `FileTrigger` (#29265)
 add a671561b8e shorten other wait times in sys tests (#29254)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/amazon/aws/operators/rds.py   | 6 +-
 tests/system/providers/amazon/aws/example_batch.py  | 2 ++
 .../providers/amazon/aws/example_eks_with_fargate_in_one_step.py| 1 +
 .../system/providers/amazon/aws/example_eks_with_fargate_profile.py | 2 ++
 .../providers/amazon/aws/example_eks_with_nodegroup_in_one_step.py  | 2 ++
 tests/system/providers/amazon/aws/example_eks_with_nodegroups.py| 2 ++
 tests/system/providers/amazon/aws/example_emr.py| 1 +
 tests/system/providers/amazon/aws/example_emr_eks.py| 2 ++
 tests/system/providers/amazon/aws/example_emr_serverless.py | 3 +++
 tests/system/providers/amazon/aws/example_glue.py   | 1 +
 tests/system/providers/amazon/aws/example_rds_export.py | 1 +
 11 files changed, 22 insertions(+), 1 deletion(-)

[GitHub] [airflow] o-nikolas merged pull request #29254: shorten other wait times in sys tests

2023-01-31 Thread via GitHub



o-nikolas merged PR #29254:
URL: https://github.com/apache/airflow/pull/29254


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #28745: annotations in logs pvc

2023-01-31 Thread via GitHub



amoghrajesh commented on issue #28745:
URL: https://github.com/apache/airflow/issues/28745#issuecomment-1410862961

   @do-it-tim added support for this in 
https://github.com/apache/airflow/pull/29270. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh opened a new pull request, #29270: Adding possibility for annotations in logs pvc

2023-01-31 Thread via GitHub



amoghrajesh opened a new pull request, #29270:
URL: https://github.com/apache/airflow/pull/29270

   
   Logs persistent volume claim doesn't have the support to add annotations 
through the helm charts. The PR adds support for the same, similar to 
https://github.com/apache/airflow/blob/main/chart/templates/dags-persistent-volume-claim.yaml#L32-L35.
 
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on pull request #29214: Helm deployment fails when postgresql.nameOverride is used

2023-01-31 Thread via GitHub



amoghrajesh commented on PR #29214:
URL: https://github.com/apache/airflow/pull/29214#issuecomment-1410854555

   Just fixed all the test case issues. @potiuk / @jedcunningham can someone 
merge the PR once the CI passes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #29226: Implement new filtering options in graph view

2023-01-31 Thread via GitHub



r-richmond commented on PR #29226:
URL: https://github.com/apache/airflow/pull/29226#issuecomment-1410852859

   Thanks for tackling this @yxiao1996. Left a tardy UI mockup suggestion for 
consideration in 
https://github.com/apache/airflow/issues/28847#issuecomment-1410851618.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] r-richmond commented on issue #28847: Graph UI: Add Filter Downstream & Filter DownStream & Upstream

2023-01-31 Thread via GitHub



r-richmond commented on issue #28847:
URL: https://github.com/apache/airflow/issues/28847#issuecomment-1410851618

   Hey @yxiao1996 this looks great. 2 questions for you
   
   1) For your example `After filtering downstream for one of the tasks`  which 
task_id did you click on & which option did you select?
   2) If you select either option and navigate to the grid view dues the filter 
on tasks remain? (It does this currently).
   
   Then lastly as a weak suggestion that can be disregarded due to my tardiness 
in getting back to you. Would it be worth keeping the same style of interface 
as the Task Actions? See my rough mockup below (note: the additional 
functionality provided with the recursive option)
   
   
   https://user-images.githubusercontent.com/9246654/215846382-68b33ed7-b14f-400c-a00d-43f8a741d466.png";>
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on issue #26725: Refactor `S3KeysUnchangedSensor` system test

2023-01-31 Thread via GitHub



vandonr-amz commented on issue #26725:
URL: https://github.com/apache/airflow/issues/26725#issuecomment-1410828693

   Yes, #28528 was about not rescheduling immediately and respecting the set 
interval, but sensors still run in reschedule mode, so this one still doesn't 
work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (51d96334b8 -> 42dd812021)

2023-01-31 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 51d96334b8 Use `schedule` param rather than `timetable` in Timetables 
docs (#29255)
 add 42dd812021 Add `FileTrigger` (#29265)

No new revisions were added by this update.

Summary of changes:
 airflow/triggers/file.py| 73 +
 tests/triggers/test_file.py | 64 +++
 2 files changed, 137 insertions(+)
 create mode 100644 airflow/triggers/file.py
 create mode 100644 tests/triggers/test_file.py

[GitHub] [airflow] kaxil merged pull request #29265: Add `FileTrigger`

2023-01-31 Thread via GitHub



kaxil merged PR #29265:
URL: https://github.com/apache/airflow/pull/29265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] syun64 commented on pull request #29093: [WIP] Enable tags for statsd metrics in a backward compatible way

2023-01-31 Thread via GitHub



syun64 commented on PR #29093:
URL: https://github.com/apache/airflow/pull/29093#issuecomment-1410802425

   > > Another alternative to this PR, is just to simply publish duplicate 
metrics
   > 
   > I was thinking exactly this when I opened this PR. Is there a good way to 
deprecate stat names? Keeping the old names forever is probably not a very 
viable solution.
   
   First off: thank you for the review! Very much appreciate you taking a look. 
I agree with you @uranusjr, that tagless metrics are very limiting for obvious 
reasons. But as I understand, there are a couple of factors that act against us 
completely deprecating the Statsd old stat names...
   
   Firstly, it looks like the long-term vision is to deprecate Statsd support 
overall and move towards adopting OpenTelemetry. And a part of that proposal is 
to deprecate Statsd metrics / the existing stat names and introducing tagged 
metrics. But as I have pointed out in this issue #11463, adopting OpenTelemetry 
will be an organizational effort rather than an individual one for users in 
large companies. On top of that, I believe that that AIP is still a long way 
from being implemented.
   
   Unfortunately with Statsd, although it is widely used, we have to deal with 
the unfortunate lack of standardization for tag-supported plugins. As an 
Influxdb-Statsd plugin user, I would love to just simplify the existing stat 
naming conventions to be more aggregation friendly and just introduce 
meaningful tags. But completely removing variable (dag, pool, etc) information 
from the existing stats metrics would be a pretty destructive change to current 
classic statsd users that may rely on that information.
   
   I think if we want to reserve that destructive change until OpenTelemetry is 
introduced, then preserving backward compatibility as proposed in this PR might 
be a gentle way to bridge the gap.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29058: Add command to export purged records to CSV files

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29058:
URL: https://github.com/apache/airflow/pull/29058#discussion_r1092263564


##
docs/apache-airflow/howto/usage-cli.rst:
##
@@ -217,6 +217,17 @@ You can use the ``--dry-run`` option to print the row 
counts in the primary tabl
 
 By default, ``db clean`` will archive purged rows in tables of the form 
``_airflow_deleted``.  If you don't want the data 
preserved in this way, you may supply argument ``--skip-archive``.
 
+Export the purged records from the archive tables
+-
+The ``db export-cleaned`` command exports the contents of the archived tables, 
created by the ``db clean`` command,
+to a specified format, by default to a CSV file. The exported file will 
contain the records that were purged from the
+primary tables during the ``db clean`` process.
+
+You can specify the export format using ``--export-format`` option. The 
default format is csv.

Review Comment:
   Ah, I get what @norm  and what you also mean: There's no default in code but 
here. Thanks, I will update it and update here too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29058: Add command to export purged records to CSV files

2023-01-31 Thread via GitHub



jedcunningham commented on code in PR #29058:
URL: https://github.com/apache/airflow/pull/29058#discussion_r1092214629


##
docs/apache-airflow/howto/usage-cli.rst:
##
@@ -217,6 +217,17 @@ You can use the ``--dry-run`` option to print the row 
counts in the primary tabl
 
 By default, ``db clean`` will archive purged rows in tables of the form 
``_airflow_deleted``.  If you don't want the data 
preserved in this way, you may supply argument ``--skip-archive``.
 
+Export the purged records from the archive tables
+-
+The ``db export-cleaned`` command exports the contents of the archived tables, 
created by the ``db clean`` command,
+to a specified format, by default to a CSV file. The exported file will 
contain the records that were purged from the
+primary tables during the ``db clean`` process.
+
+You can specify the export format using ``--export-format`` option. The 
default format is csv.

Review Comment:
   Let me rephrase, as I have the same question (I think).
   
   ``export-format`` isn't required, but also has no default. So you must pass 
`csv`, and it's also the only option.
   
   I think having `csv` be the actual default is okay, but we need to mark it 
as such (then we wouldn't need to mark it as required). We should also list out 
the valid options here, which is only csv today.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yxiao1996 commented on pull request #29226: Implement new filtering options in graph view

2023-01-31 Thread via GitHub



yxiao1996 commented on PR #29226:
URL: https://github.com/apache/airflow/pull/29226#issuecomment-1410730706

   looks like it times out waiting for CI image
   
   ```
  Progress: Image 3.7 Waiting: #675 
ghcr.io/apache/airflow/main/ci/python3.7:627367d641c8eec38e6209a68ae16a7af813bb59.
 
─
 Time passed: 01:59:29 
──
 Error: The operation was canceled.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29248: Release notes for helm chart 1.8.0

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29248:
URL: https://github.com/apache/airflow/pull/29248#discussion_r1092200171


##
chart/RELEASE_NOTES.rst:
##
@@ -23,15 +23,116 @@ Run ``helm repo update`` before upgrading the chart to the 
latest version.
 
 .. towncrier release notes start
 
+Airflow Helm Chart 1.8.0 (2023-02-02)
+-
+
+Significant Changes
+^^^
+
+``bitnami/postgresql`` subchart updated to ``12.1.9`` (#29071)
+""
+
+The version of postgresql installed is still version 11.
+
+If you are upgrading an existing helm release with the built-in postgres 
database, you will either need to delete your release and reinstall fresh, or 
manually delete these 2 objects:
+
+```
+kubectl delete secret {RELEASE_NAME}-postgresql
+kubectl delete statefulset {RELEASE_NAME}-postgresql
+```
+
+As a reminder, it is recommended to `set up an external database 
`_
 in production.
+
+This version of the chart uses different variable names for setting usernames 
and passwords in the postgres database.
+
+- ``postgresql.auth.enablePostgresUser`` is used to determine if the 
"postgres" admin account will be created.
+- ``postgresql.auth.postgresPassword`` sets the password for the "postgres" 
user.
+- ``postgresql.auth.username`` and ``postrgesql.auth.password`` are used to 
set credentials for a non-admin account if desired.
+- ``postgresql.postgresqlUsername`` and ``postgresql.postresqlPassword``, 
which were used in the previous version of the chart, are no longer used.
+
+Users will need to make those changes in their values files if they are 
changing the Postgres configuration.
+
+Previously the subchart version was ``10.5.3``.
+
+Default ``dags.gitSync.wait`` reduced to ``5`` seconds (#27625)
+"""
+
+The default for ``dags.gitSync.wait`` has been reduced from ``60`` seconds to 
``5`` seconds to reduce the likelihood of DAGs
+becoming inconsistent between Airflow components. This will, however, increase 
traffic to the remote git repository.
+
+Default Airflow image is updated to ``2.5.1`` (#29074)
+""
+
+The default Airflow image that is used with the Chart is now ``2.5.1``, 
previously it was ``2.4.1``.
+
+Default git-sync image is updated to ``3.6.3`` (#27848)
+"""
+
+The default git-sync image that is used with the Chart is now ``3.6.3``, 
previously it was ``3.4.0``.
+
+Default redis image is updated to ``7-bullseye`` (#27443)
+"
+
+The default redis image that is used with the Chart is now ``7-bullseye``, 
previously it was ``6-bullseye``.
+
+New Features
+
+
+- Add annotations on deployments (#28688)
+- Add global volume & volumeMounts to the chart (#27781)
+
+Improvements
+
+
+- Add support for ``webserverConfigConfigMapName`` (#27419)
+- Enhance chart to allow over-riding command-line args to statsd exporter 
(#28041)

Review Comment:
   ```suggestion
   - Enhance chart to allow overriding command-line args to statsd exporter 
(#28041)
   ```



##
chart/Chart.yaml:
##
@@ -67,187 +67,143 @@ annotations:
 - title: Task Instance Context Menu
   url: 
https://airflow.apache.org/docs/apache-airflow/2.5.1/_images/context.png
   artifacthub.io/changes: |
-- description: Make cleanup job history configurable
+- description: Add annotations on deployments
   kind: added
   links:
-  - name: '#26838'
-url: https://github.com/apache/airflow/pull/26838
-- description: Added labels to specific Airflow components
+  - name: '#28688'
+url: https://github.com/apache/airflow/pull/28688
+- description: Add global volume & volumeMounts to the chart
   kind: added
   links:
-  - name: '#25031'
-url: https://github.com/apache/airflow/pull/25031
-- description: Add StatsD ``overrideMappings`` in Helm chart values
-  kind: added
-  links:
-  - name: '#26598'
-url: https://github.com/apache/airflow/pull/26598
-- description: Adding ``podAnnotations`` to StatsD deployment template
-  kind: added
-  links:
-  - name: '#25732'
-url: https://github.com/apache/airflow/pull/25732
-- description: Container specific extra environment variables
-  kind: added
-  links:
-  - name: '#24784'
-url: https://github.com/apache/airflow/pull/24784
-- description: Custom labels for extrasecrets/configmaps
-  kind: added
-  links:
-  - name: '#25283'
-url: https://github.com/apache/airflow/pull/25283
-- description: Add ``revisionHistoryLimit`` to all deployments
-  kind: added
-  links:
-  - name: '#2505

[GitHub] [airflow] kaxil commented on a diff in pull request #29265: Add `FileTrigger`

2023-01-31 Thread via GitHub



kaxil commented on code in PR #29265:
URL: https://github.com/apache/airflow/pull/29265#discussion_r1092166719


##
tests/triggers/test_file.py:
##
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+
+from airflow.triggers.file import FileTrigger
+
+
+class TestFileTrigger:
+FILE_PATH = "/files/dags/example_async_file.py"
+
+def test_serialization(self):
+"""Asserts that the trigger correctly serializes its arguments and 
classpath."""
+trigger = FileTrigger(filepath=self.FILE_PATH, poll_interval=5)
+classpath, kwargs = trigger.serialize()
+assert classpath == "airflow.triggers.file.FileTrigger"
+assert kwargs == {
+"filepath": self.FILE_PATH,
+"poll_interval": 5,
+"recursive": False,
+}
+
+@pytest.mark.asyncio
+async def test_task_file_trigger(self, tmp_path):

Review Comment:
   Check L52 that does that :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] waldoppper opened a new issue, #29268: Statsd metric name components are unsanitized

2023-01-31 Thread via GitHub



waldoppper opened a new issue, #29268:
URL: https://github.com/apache/airflow/issues/29268

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   [2.2.5]
   I recently enabled statsd metric emission on my team's deploy and began 
writing views on the result. 
   
   I've found that:
   1. some dag authors had included a period in their task_id name - like 
`task_id = "load.all"`
   2. this string is not sanitized of statsd-sensitive characters (like '.' and 
'|') prior to the [building of the Metric 
name](https://github.com/apache/airflow/blob/main/airflow/models/taskinstance.py#L1504)
   
   Result: metric names with too many .-delimiters and grafana views which 
don't render these task metrics. 
   
   ### What you think should happen instead
   
   Any instance of `Statsd.foo(f'{dag_id}...')` or 
`Statsd.foo(f'{task_id}...')` should probably first sanitize those strings of 
statsd-sensitive characters. 
   
   So instead of
   
   Current: 
`Statsd.timer("dag.{self.task.dag_id}.{self.task.task_id}.duration")` 
   
   Recommended: 
`Stats.timer(f"dag.{statsd_sanitize(self.task.dag_id)}.{statsd_sanitize(self.task.task_id)}.duration")`
   
   ### How to reproduce
   
   1. create a dag
   2. write a task in that dag with task_id which contains a period. Like: 
"foo.bar"
   3. enable statsd metrics collection
   4. run your dag
   5. struggle to deal with poorly-formatted metric names like 
`"dag.DAG.WITH.PERIODS.IN.NAME.TASK.WITH.PERIODS.IN.NAME.duration"`, the client 
emits `"dag.DAG_WITH_PERIODS_IN_NAME.TASK_WITH_PERIODS_IN_NAME.duration"` 
   
   ### Operating System
   
   debian
   
   ### Versions of Apache Airflow Providers
   
   ```apache-airflow-providers-amazon==3.2.0
   apache-airflow-providers-apache-hive==2.3.1
   apache-airflow-providers-apache-spark==2.1.2
   apache-airflow-providers-celery==2.1.3
   apache-airflow-providers-cncf-kubernetes==3.0.0
   apache-airflow-providers-docker==2.5.2
   apache-airflow-providers-elasticsearch==2.2.0
   apache-airflow-providers-ftp==2.1.2
   apache-airflow-providers-google==6.7.0
   apache-airflow-providers-grpc==2.0.4
   apache-airflow-providers-hashicorp==2.1.4
   apache-airflow-providers-http==2.1.2
   apache-airflow-providers-imap==2.2.3
   apache-airflow-providers-microsoft-azure==3.7.2
   apache-airflow-providers-mysql==2.2.3
   apache-airflow-providers-odbc==2.0.4
   apache-airflow-providers-postgres==4.1.0
   apache-airflow-providers-redis==2.0.4
   apache-airflow-providers-sendgrid==2.0.4
   apache-airflow-providers-sftp==2.5.2
   apache-airflow-providers-slack==4.2.3
   apache-airflow-providers-sqlite==2.1.3
   apache-airflow-providers-ssh==2.4.3
   ```
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Deployed with v8.6.1 of 
https://github.com/airflow-helm/charts/tree/main/charts/airflow 
   
   We wired up a pod to convert statsd->prom using 
https://github.com/prometheus/statsd_exporter, and use Matcher rules by 
glob/regex to convert statsd-style metrics into prom-style. 
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29268: Statsd metric name components are unsanitized

2023-01-31 Thread boring-cyborg



boring-cyborg[bot] commented on issue #29268:
URL: https://github.com/apache/airflow/issues/29268#issuecomment-1410653574

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #29243: Auto refresh DAG state in UI

2023-01-31 Thread via GitHub



bbovenzi commented on issue #29243:
URL: https://github.com/apache/airflow/issues/29243#issuecomment-1410650289

   I'm going to remove the "good first issue". It's not a trivial change.
   
   That `state_token` is defined by Flask App Builder and not in js, and the 
auto-refresh in the graph view currently only checks for new task instances, 
not the dag run state. So we'd have to add in another endpoint to refresh and 
then update the ``. 
   
   Meanwhile, I'm actively working on a new graph view for 2.6:
   https://user-images.githubusercontent.com/4600967/215813495-8710b2cb-7bc3-4724-8b84-6aa6a7bc75a2.png";>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29058: Add command to export purged records to CSV files

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29058:
URL: https://github.com/apache/airflow/pull/29058#discussion_r1092132580


##
airflow/utils/db_cleanup.py:
##
@@ -123,6 +127,15 @@ def _check_for_rows(*, query: Query, print_rows=False):
 return num_entities
 
 
+def _dump_table_to_file(*, target_table, file_path, export_format, session):
+if export_format == "csv":

Review Comment:
   Actually, this is for future works that would add extra format options, so 
for json, we could say `if export_format=="json"`. we have protection on the 
choices using argparse `choices`. Do you suggest I add the message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29058: Add command to export purged records to CSV files

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29058:
URL: https://github.com/apache/airflow/pull/29058#discussion_r1092120446


##
docs/apache-airflow/howto/usage-cli.rst:
##
@@ -217,6 +217,17 @@ You can use the ``--dry-run`` option to print the row 
counts in the primary tabl
 
 By default, ``db clean`` will archive purged rows in tables of the form 
``_airflow_deleted``.  If you don't want the data 
preserved in this way, you may supply argument ``--skip-archive``.
 
+Export the purged records from the archive tables
+-
+The ``db export-cleaned`` command exports the contents of the archived tables, 
created by the ``db clean`` command,
+to a specified format, by default to a CSV file. The exported file will 
contain the records that were purged from the
+primary tables during the ``db clean`` process.
+
+You can specify the export format using ``--export-format`` option. The 
default format is csv.

Review Comment:
   We want to add more options in the future, so I chose default to be future 
proof



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] VladaZakharova commented on pull request #29266: Add deferrable mode to GKEStartPodOperator

2023-01-31 Thread via GitHub



VladaZakharova commented on PR #29266:
URL: https://github.com/apache/airflow/pull/29266#issuecomment-1410585964

   @potiuk @kaxil 
   Hi!
   Can you please check the changes for the GKEStartPod operator, please? This 
is the second part of the changes from the original PR. Many thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] tsoud commented on a diff in pull request #29158: Update pipeline.rst - Fix query in `merge_data()` task

2023-01-31 Thread via GitHub



tsoud commented on code in PR #29158:
URL: https://github.com/apache/airflow/pull/29158#discussion_r1092087567


##
docs/apache-airflow/tutorial/pipeline.rst:
##
@@ -170,7 +170,7 @@ Here we select completely unique records from the retrieved 
data, then we check
   FROM employees_temp
   ) t
   ON CONFLICT ("Serial Number") DO UPDATE
-  SET "Serial Number" = excluded."Serial Number";
+  SET "Serial Number" = t."Serial Number";

Review Comment:
   @josh-fell, thanks for the feedback! 
   
   Yep, that definitely makes much more sense. "Serial Number" is a PK so it's 
the other fields should be updated on conflict.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BasPH commented on pull request #29143: Demonstrate usage of the PythonSensor

2023-01-31 Thread via GitHub



BasPH commented on PR #29143:
URL: https://github.com/apache/airflow/pull/29143#issuecomment-1410573849

   I suggest showing reproducible examples. The current example for the 
PythonSensor is not easily reproducible because it doesn't show imports.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #28745: annotations in logs pvc

2023-01-31 Thread via GitHub



amoghrajesh commented on issue #28745:
URL: https://github.com/apache/airflow/issues/28745#issuecomment-1410523257

   I can take it up then. @potiuk can I?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29224: add output format arg for `cli.dags.trigger`

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29224:
URL: https://github.com/apache/airflow/pull/29224#discussion_r1092054686


##
airflow/api/client/local_client.py:
##
@@ -22,21 +22,39 @@
 from airflow.api.common import delete_dag, trigger_dag
 from airflow.api.common.experimental.get_lineage import get_lineage as 
get_lineage_api
 from airflow.exceptions import AirflowBadRequest, PoolNotFound
+from airflow.models import DagRun
 from airflow.models.pool import Pool
 
 
 class Client(api_client.Client):
 """Local API client implementation."""
 
-def trigger_dag(self, dag_id, run_id=None, conf=None, execution_date=None, 
replace_microseconds=True):
+def trigger_dag(
+self, dag_id, run_id=None, conf=None, execution_date=None, 
replace_microseconds=True
+) -> dict | None:
 dag_run = trigger_dag.trigger_dag(
 dag_id=dag_id,
 run_id=run_id,
 conf=conf,
 execution_date=execution_date,
 replace_microseconds=replace_microseconds,
 )
-return f"Created {dag_run}"
+if isinstance(dag_run, DagRun):

Review Comment:
   ```suggestion
   if dag_run:
   ```
   Do we need the isinstance?
   Also let's add a test for this change in 
tests/api/client/test_local_client.py



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] do-it-tim commented on issue #28745: annotations in logs pvc

2023-01-31 Thread via GitHub



do-it-tim commented on issue #28745:
URL: https://github.com/apache/airflow/issues/28745#issuecomment-1410489399

   yes, think that would be the equivalent way to do it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #29265: Add `FileTrigger`

2023-01-31 Thread via GitHub



ephraimbuddy commented on code in PR #29265:
URL: https://github.com/apache/airflow/pull/29265#discussion_r1092031302


##
tests/triggers/test_file.py:
##
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+
+from airflow.triggers.file import FileTrigger
+
+
+class TestFileTrigger:
+FILE_PATH = "/files/dags/example_async_file.py"
+
+def test_serialization(self):
+"""Asserts that the trigger correctly serializes its arguments and 
classpath."""
+trigger = FileTrigger(filepath=self.FILE_PATH, poll_interval=5)
+classpath, kwargs = trigger.serialize()
+assert classpath == "airflow.triggers.file.FileTrigger"
+assert kwargs == {
+"filepath": self.FILE_PATH,
+"poll_interval": 5,
+"recursive": False,
+}
+
+@pytest.mark.asyncio
+async def test_task_file_trigger(self, tmp_path):

Review Comment:
   Should we test the implementation of the run function? Looks like we only 
tested the behaviour



##
airflow/triggers/file.py:
##
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+import datetime
+import os
+import typing
+from glob import glob
+from typing import Any
+
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+
+class FileTrigger(BaseTrigger):
+"""
+A trigger that fires exactly once after it finds the requested file or 
folder.
+
+:param filepath: File or folder name (relative to the base path set within 
the connection), can
+be a glob.
+:param recursive: when set to ``True``, enables recursive directory 
matching behavior of
+``**`` in glob filepath parameter. Defaults to ``False``.
+"""
+
+def __init__(
+self,
+filepath: str,
+recursive: bool = False,
+poll_interval: float = 5.0,
+):
+super().__init__()
+self.filepath = filepath
+self.recursive = recursive
+self.poll_interval = poll_interval
+
+def serialize(self) -> tuple[str, dict[str, Any]]:
+"""Serializes FileTrigger arguments and classpath."""
+return (
+"airflow.triggers.file.FileTrigger",
+{
+"filepath": self.filepath,
+"recursive": self.recursive,
+"poll_interval": self.poll_interval,
+},
+)
+
+async def run(self) -> typing.AsyncIterator["TriggerEvent"]:
+"""Simple loop until the relevant files are found."""

Review Comment:
   ```suggestion
   """Loop until the relevant files are found."""
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] VladaZakharova closed pull request #29266: Add deferrable mode to GKEStartPodOperator

2023-01-31 Thread via GitHub



VladaZakharova closed pull request #29266: Add deferrable mode to 
GKEStartPodOperator
URL: https://github.com/apache/airflow/pull/29266


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #28745: annotations in logs pvc

2023-01-31 Thread via GitHub



amoghrajesh commented on issue #28745:
URL: https://github.com/apache/airflow/issues/28745#issuecomment-1410470039

   Oh, thanks @do-it-tim. It would be nice to accept it from: 
`.Values.logs.persistence.annotations`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on issue #28745: annotations in logs pvc

2023-01-31 Thread via GitHub



amoghrajesh commented on issue #28745:
URL: https://github.com/apache/airflow/issues/28745#issuecomment-1410454908

   @potiuk @do-it-tim if nobody is actively developing this, i would be more 
than happy to take it up. I would require some more context and concrete idea 
before starting though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell commented on pull request #28721: Use connection URI in SqliteHook

2023-01-31 Thread via GitHub



josh-fell commented on PR #28721:
URL: https://github.com/apache/airflow/pull/28721#issuecomment-1410452249

   > LGTM. Does it need docs 
https://airflow.apache.org/docs/apache-airflow-providers-sqlite/stable/connections/sqlite.html
 updating?
   
   +1. [There is a PR 
in-flight](https://github.com/apache/airflow/pull/29139#pullrequestreview-1268619921)
 which was planning to update the docs around connecting to SQLite in an 
attempt [to fix this issue](https://github.com/apache/airflow/issues/29113). 
Not sure how we typical handle this collision. Doing a single update in a 
commit would be best.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BasPH opened a new issue, #29267: Support tabs in docs

2023-01-31 Thread via GitHub



BasPH opened a new issue, #29267:
URL: https://github.com/apache/airflow/issues/29267

   ### What do you see as an issue?
   
   I suggest supporting tabs in the docs to improve the readability when 
demonstrating different ways to achieve the same things.
   
   **Motivation**
   We have multiple ways to achieve the same thing in Airflow, for example:
   
   - TaskFlow API & "classic" operators
   - CLI & REST API & API client
   
   However, our docs currently do not consistently demonstrate different ways 
to use Airflow. For example, 
https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html
 demonstrates TaskFlow operators in some examples and classic operators in 
other examples. All cases covered can be supported by both the TaskFlow & 
classic operators.
   
   In the case of 
https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html,
 I think a nice solution to demonstrate both approaches would be to use tabs. 
That way somebody who prefers the TaskFlow API can view all TaskFlow examples, 
and somebody who prefers the classic operators (we should give those a better 
name) can view only those examples.
   
   **Possible implementation**
   There is a package 
[sphinx-tabs](https://github.com/executablebooks/sphinx-tabs) for this. For the 
example above, having https://sphinx-tabs.readthedocs.io/en/latest/#group-tabs 
would be great because it enables you to view all examples of one "style" with 
a single click.
   
   ### Solving the problem
   
   Install https://github.com/executablebooks/sphinx-tabs with the docs.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nsphung commented on issue #29112: "Operation not permitted" error when chmod on log folder

2023-01-31 Thread via GitHub



nsphung commented on issue #29112:
URL: https://github.com/apache/airflow/issues/29112#issuecomment-1410442324

   I have the same issue with image based on `apache/airflow:2.5.1-python3.9`. 
I have updated due to CVE-2023-22884


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] VladaZakharova opened a new pull request, #29266: Add deferrable mode to GKEStartPodOperator

2023-01-31 Thread via GitHub



VladaZakharova opened a new pull request, #29266:
URL: https://github.com/apache/airflow/pull/29266

   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request, #29265: Add `FileTrigger`

2023-01-31 Thread via GitHub



kaxil opened a new pull request, #29265:
URL: https://github.com/apache/airflow/pull/29265

   Contributes back one of the core Triggers from 
https://github.com/astronomer/astronomer-providers so that it can be used to 
create an operator /sensor or used within taskflow API
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] norm commented on a diff in pull request #29058: Add command to export purged records to CSV files

2023-01-31 Thread via GitHub



norm commented on code in PR #29058:
URL: https://github.com/apache/airflow/pull/29058#discussion_r1091933569


##
docs/apache-airflow/howto/usage-cli.rst:
##
@@ -217,6 +217,17 @@ You can use the ``--dry-run`` option to print the row 
counts in the primary tabl
 
 By default, ``db clean`` will archive purged rows in tables of the form 
``_airflow_deleted``.  If you don't want the data 
preserved in this way, you may supply argument ``--skip-archive``.
 
+Export the purged records from the archive tables
+-
+The ``db export-cleaned`` command exports the contents of the archived tables, 
created by the ``db clean`` command,
+to a specified format, by default to a CSV file. The exported file will 
contain the records that were purged from the
+primary tables during the ``db clean`` process.
+
+You can specify the export format using ``--export-format`` option. The 
default format is csv.

Review Comment:
   Default … or only? If you fake add `json` as a second option, is `csv` still 
chosen by default?



##
airflow/utils/db_cleanup.py:
##
@@ -123,6 +127,15 @@ def _check_for_rows(*, query: Query, print_rows=False):
 return num_entities
 
 
+def _dump_table_to_file(*, target_table, file_path, export_format, session):
+if export_format == "csv":

Review Comment:
   It's a minor thing since you *shouldn't* be able to get here without `csv` 
being the format, but there's no code to catch what-if-no-acceptable-format.
   
   (I mean, "you shouldn't see this" is a classic comedy error message for a 
reason, someone will get there)



##
docs/apache-airflow/howto/usage-cli.rst:
##
@@ -217,6 +217,17 @@ You can use the ``--dry-run`` option to print the row 
counts in the primary tabl
 
 By default, ``db clean`` will archive purged rows in tables of the form 
``_airflow_deleted``.  If you don't want the data 
preserved in this way, you may supply argument ``--skip-archive``.
 
+Export the purged records from the archive tables
+-
+The ``db export-cleaned`` command exports the contents of the archived tables, 
created by the ``db clean`` command,
+to a specified format, by default to a CSV file. The exported file will 
contain the records that were purged from the
+primary tables during the ``db clean`` process.
+
+You can specify the export format using ``--export-format`` option. The 
default format is csv.
+
+You must also specify the location of the path to which you want to export the 
data using ``--output-path`` option.

Review Comment:
   Worth mentioning that the dir must exist here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

1 2 >

1 - 100 of 140 matches

Mail list logo