[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'

[GitBox]

patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134202626

   @dstandish are you asking just for import statement? If yes then here it is:
   ```
   from kubernetes.client.models import (
   V1Container,
   V1ObjectMeta,
   V1Pod,
   V1PodSpec,
   V1ResourceRequirements,
   )
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] msumit opened a new pull request, #23861: Add default task retry delay config

[GitBox]

msumit opened a new pull request, #23861:
URL: https://github.com/apache/airflow/pull/23861

   Adding a cluster level support for task retry delay, so one doesn't need to 
specify at each dag or task level if it's different from 300s. 
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragement file, named `{pr_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] akakakakakaa commented on pull request #23531: make resources for KubernetesPodTemplate templated

[GitBox]

akakakakakaa commented on PR #23531:
URL: https://github.com/apache/airflow/pull/23531#issuecomment-1134181785

   In addition to previously mentioned by @lior1990,
   
   ```
   class CustomKubernetesPodOperator(KubernetesPodOperator):
   template_fields: Sequence[str] = KubernetesPodOperator.template_fields + 
('k8s_resources', )
   
   def _render_nested_template_fields(
   self,
   content: Any,
   context: 'Context',
   jinja_env: "jinja2.Environment",
   seen_oids: set,
   ) -> None:
   if id(content) not in seen_oids and isinstance(content, 
k8s.V1ResourceRequirements):
   seen_oids.add(id(content))
   self._do_render_template_fields(content, ("limits", "requests"), 
context, jinja_env, seen_oids)
   return
   
   super()._render_nested_template_fields(content, context, jinja_env, 
seen_oids)
   ```
   
   In KubernetesPodOperator, resources allocated to k8s_resources
   ```
   self.k8s_resources = convert_resources(resources) if resources else {}
   
   Instead of resource, k8s_resource should be added to template_fields.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] akakakakakaa commented on pull request #23531: make resources for KubernetesPodTemplate templated

[GitBox]

akakakakakaa commented on PR #23531:
URL: https://github.com/apache/airflow/pull/23531#issuecomment-1134177937

   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] tanelk commented on a diff in pull request #21877: AIP-45 Remove dag parsing in airflow run local

[GitBox]

tanelk commented on code in PR #21877:
URL: https://github.com/apache/airflow/pull/21877#discussion_r879012803


##
airflow/jobs/scheduler_job.py:
##
@@ -764,6 +765,26 @@ def _execute(self) -> None:
 self.log.exception("Exception when executing 
DagFileProcessorAgent.end")
 self.log.info("Exited execute loop")
 
+def _update_dag_run_state_for_paused_dags(self):
+try:
+paused_dag_ids = DagModel.get_all_paused_dag_ids()
+for dag_id in paused_dag_ids:
+if dag_id in self._paused_dag_without_running_dagruns:
+continue

Review Comment:
   dag_ids are never removed from this set. If I unpause the DAG after this has 
run and then later re-pause it, then this method skips it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23510: Fix permission issue for dag that has dot in name

[GitBox]

github-actions[bot] commented on PR #23510:
URL: https://github.com/apache/airflow/pull/23510#issuecomment-1134136285

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #23720: Fix backfill queued task getting reset to scheduled state.

[GitBox]

uranusjr commented on code in PR #23720:
URL: https://github.com/apache/airflow/pull/23720#discussion_r878998038


##
airflow/executors/kubernetes_executor.py:
##
@@ -530,13 +532,15 @@ def start(self) -> None:
 self.kube_config.worker_pods_pending_timeout_check_interval,
 self._check_worker_pods_pending_timeout,
 )
-self.event_scheduler.call_regular_interval(
-self.kube_config.worker_pods_queued_check_interval,
-self.clear_not_launched_queued_tasks,
-)
-# We also call this at startup as that's the most likely time to see
-# stuck queued tasks
-self.clear_not_launched_queued_tasks()
+
+if self.job_id != 'manual':

Review Comment:
   Any reason why this excludes `manual`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #23530: Dagrun run TaskInstance API

[GitBox]

uranusjr commented on code in PR #23530:
URL: https://github.com/apache/airflow/pull/23530#discussion_r878983519


##
airflow/api_connexion/endpoints/task_instance_endpoint.py:
##
@@ -512,3 +517,86 @@ def post_set_task_instances_state(*, dag_id: str, session: 
Session = NEW_SESSION
 session=session,
 )
 return 
task_instance_reference_collection_schema.dump(TaskInstanceReferenceCollection(task_instances=tis))
+
+
+@security.requires_access(
+[
+(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
+(permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG_RUN),
+(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_TASK_INSTANCE),
+],
+)
+@provide_session
+def run_task_instance(
+*, dag_id: str, dag_run_id: str, task_id: str, session: Session = 
NEW_SESSION

Review Comment:
   The `session` value should be used much more in this function. There are at 
least two functions I can see are creating a new session for no obvious reason.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23803: More flexible dynamic task mapping

[GitBox]

uranusjr commented on issue #23803:
URL: https://github.com/apache/airflow/issues/23803#issuecomment-1134107753

   The main problem of using zipping against a classic operator class is not 
really about the zip operation itself, but to unpack the zipped iterator (i.e. 
the `arg1, arg2 = param` part). Even if you can produce a zip iterator (which 
you already can), it’s still difficult to use the iterator against a classic 
operator. I am yet to be able to come up with a reasonable API either; 
instantiating a class instance is (kind of) fundamentally not straightforward 
in a unctional programming context.
   
   Note that you can also already kind of do this like below. There’re a lot of 
caveats on calling `execute` directly, but it’s kind of doable. (Not that I 
particularly recommend it though.)
   
   ```
   @task
   def run_task(param):
   image, arguments = param
   op = KubernetesPodOperator(task_id="my-task", image=image, 
arguments=arguments, ...)
   op.execute()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #22854: Render list items in rendered fields view

[GitBox]

uranusjr commented on code in PR #22854:
URL: https://github.com/apache/airflow/pull/22854#discussion_r878979381


##
airflow/www/utils.py:
##
@@ -483,20 +483,37 @@ def pygment_html_render(s, lexer=lexers.TextLexer):
 return highlight(s, lexer(), HtmlFormatter(linenos=True))
 
 
-def render(obj, lexer):
+def render(obj: Any, lexer, handler=None):
 """Render a given Python object with a given Pygments lexer"""
-out = ""
+
 if isinstance(obj, str):
-out = Markup(pygment_html_render(obj, lexer))
-elif isinstance(obj, (tuple, list)):
+return Markup(pygment_html_render(obj, lexer))
+
+if isinstance(obj, (tuple, list)):
+out = ""

Review Comment:
   Instead of repeatedly call `+` on strings, it may be a better idea to turn 
this into a generator, and do a `join` instead.
   
   ```python
   def _render_parts(obj, lexer, handler):
   ...
   yield Markup(...)
   ...
   
   def render(obj, lexer, handler):
   return "".join(_render_parts(obj, lexer, handler))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #22854: Render list items in rendered fields view

[GitBox]

uranusjr commented on code in PR #22854:
URL: https://github.com/apache/airflow/pull/22854#discussion_r878978731


##
airflow/www/utils.py:
##
@@ -483,20 +483,37 @@ def pygment_html_render(s, lexer=lexers.TextLexer):
 return highlight(s, lexer(), HtmlFormatter(linenos=True))
 
 
-def render(obj, lexer):
+def render(obj: Any, lexer, handler=None):

Review Comment:
   Would be nice to also annotate `handler`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #22854: Render list items in rendered fields view

[GitBox]

uranusjr commented on code in PR #22854:
URL: https://github.com/apache/airflow/pull/22854#discussion_r878978625


##
airflow/www/templates/airflow/ti_code.html:
##
@@ -21,10 +21,10 @@
 {% block title %}DAGs - {{ appbuilder.app_name }}{% endblock %}
 
 {% block content %}
-  {{ super() }}
-  {{ title }}
-  {% for k, v in html_dict.items() %}
-{{ k }}
-{{ v }}
-  {% endfor %}
+{{ super() }}
+{{ title }}
+{% for k, v in html_dict.items() %}
+{{ k }}

Review Comment:
   We should probably add CSS to style `h5` instead of adding a `strong`? Or 
maybe this does not really matter at this point. cc @bbovenzi 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #23846: Do not fail requeued TIs

[GitBox]

uranusjr commented on code in PR #23846:
URL: https://github.com/apache/airflow/pull/23846#discussion_r878977989


##
airflow/jobs/scheduler_job.py:
##
@@ -664,7 +663,20 @@ def _process_executor_events(self, session: Session = 
None) -> int:
 ti.pid,
 )
 
-if ti.try_number == buffer_key.try_number and ti.state == 
State.QUEUED:
+# There are two scenarios why the same TI with the same try_number 
is queued
+# after executor is finished with it:
+# 1) the TI was killed externally and it had no time to mark 
itself failed
+# - in this case we should mark it as failed here.
+# 2) the TI has been requeued after getting deferred - in this 
case either our executor has it
+# or the TI is queued by another job. Either ways we should not 
fail it.
+
+# All of this could also happen if the state is "running",
+# but that is handled by the zombie detection.
+
+ti_queued = ti.try_number == buffer_key.try_number and ti.state == 
State.QUEUED

Review Comment:
   ```suggestion
   ti_queued = ti.try_number == buffer_key.try_number and ti.state 
== TaskInstanceState.QUEUED
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23597: Clean up f-strings in logging calls

[GitBox]

github-actions[bot] commented on PR #23597:
URL: https://github.com/apache/airflow/pull/23597#issuecomment-1134084723

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #23829: Add disable retry flag on backfill

[GitBox]

uranusjr commented on code in PR #23829:
URL: https://github.com/apache/airflow/pull/23829#discussion_r878969620


##
airflow/cli/cli_parser.py:
##
@@ -339,6 +339,11 @@ def string_lower_type(val):
 help=("if set, the backfill will keep going even if some of the tasks 
failed"),
 action="store_true",
 )
+ARG_DISABLE_RETRY = Arg(
+("-dr", "--disable-retry"),

Review Comment:
   The `-dr` flag feels unnecessary, and I don’t think we have a precedence for 
multi-character short options.
   
   ```suggestion
   ("--disable-retry",),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #23829: Add disable retry flag on backfill

[GitBox]

uranusjr commented on code in PR #23829:
URL: https://github.com/apache/airflow/pull/23829#discussion_r878969191


##
airflow/jobs/backfill_job.py:
##
@@ -627,6 +629,11 @@ def to_keep(key: TaskInstanceKey) -> bool:
 for new_ti in new_mapped_tis:
 new_ti.set_state(TaskInstanceState.SCHEDULED, 
session=session)
 
+# Set state to failed for running TIs that are set up for retry if 
disable-retry flag is set
+for ti in list(ti_status.running.values()):

Review Comment:
   Why the `list()` call?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr opened a new pull request, #23860: Remove special serde logic for mapped op_kwargs

[GitBox]

uranusjr opened a new pull request, #23860:
URL: https://github.com/apache/airflow/pull/23860

   This was done in `PythonOperator` to simplify the serialization format. But 
it turns out parsing out a correct format is too much work and too easy to get 
wrong, so let's not do that.
   
   Note that this does not affect decorated taskflow operators (which use 
`mapped_op_kwargs` instead), only plain classic `PythonOperator` subclasses.
   
   This change requires all mapped operators on classic `PythonOperators` 
subclasses to be re-serialized. But this is likely a reasonable tradeoff over 
carrying complex compatibility code forever.
   
   Fix #23833. Submitting as draft to catch all needed test changes first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add __wrapped__ property to _TaskDecorator (#23830)

[uranusjr]

This is an automated email from the ASF dual-hosted git repository.

uranusjr pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a71e4b7890 Add __wrapped__ property to _TaskDecorator (#23830)
a71e4b7890 is described below

commit a71e4b789006b8f36cd993731a9fb7d5792fccc2
Author: sanjayp 
AuthorDate: Mon May 23 06:54:52 2022 +0530

Add __wrapped__ property to _TaskDecorator (#23830)

Co-authored-by: Sanjay Pillai 
---
 airflow/decorators/base.py  | 10 +-
 tests/decorators/test_python.py | 17 +
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/airflow/decorators/base.py b/airflow/decorators/base.py
index 2029f6c5ed..79277c7281 100644
--- a/airflow/decorators/base.py
+++ b/airflow/decorators/base.py
@@ -267,6 +267,10 @@ class _TaskDecorator(Generic[Function, OperatorSubclass]):
 op.doc_md = self.function.__doc__
 return XComArg(op)
 
+@property
+def __wrapped__(self) -> Function:
+return self.function
+
 @cached_property
 def function_signature(self):
 return inspect.signature(self.function)
@@ -495,6 +499,10 @@ class Task(Generic[Function]):
 
 function: Function
 
+@property
+def __wrapped__(self) -> Function:
+...
+
 def expand(self, **kwargs: "Mappable") -> XComArg:
 ...
 
@@ -527,7 +535,7 @@ def task_decorator_factory(
 **kwargs,
 ) -> TaskDecorator:
 """
-A factory that generates a wrapper that raps a function into an Airflow 
operator.
+A factory that generates a wrapper that wraps a function into an Airflow 
operator.
 Accepts kwargs for operator kwarg. Can be reused in a single DAG.
 
 :param python_callable: Function to decorate
diff --git a/tests/decorators/test_python.py b/tests/decorators/test_python.py
index 3f8b44c464..63514fcf25 100644
--- a/tests/decorators/test_python.py
+++ b/tests/decorators/test_python.py
@@ -738,3 +738,20 @@ def test_mapped_render_template_fields(dag_maker, session):
 
 assert op.op_kwargs['arg1'] == "{{ ds }}"
 assert op.op_kwargs['arg2'] == "fn"
+
+
+def test_task_decorator_has_wrapped_attr():
+"""
+Test  @task original underlying function is accessible
+through the __wrapped__ attribute.
+"""
+
+def org_test_func():
+pass
+
+decorated_test_func = task_decorator(org_test_func)
+
+assert hasattr(
+decorated_test_func, '__wrapped__'
+), "decorated function does not have __wrapped__ attribute"
+assert decorated_test_func.__wrapped__ is org_test_func, "__wrapped__ attr 
is not the original function"

[GitHub] [airflow] uranusjr merged pull request #23830: Add __wrapped__ property to _TaskDecorator

[GitBox]

uranusjr merged PR #23830:
URL: https://github.com/apache/airflow/pull/23830


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr closed issue #23688: _TaskDecorator has no __wrapped__ attribute in v2.3.0

[GitBox]

uranusjr closed issue #23688: _TaskDecorator has no __wrapped__ attribute in 
v2.3.0
URL: https://github.com/apache/airflow/issues/23688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #20461: Airflow is trying to schedule tasks prior to DAG's start_date

[GitBox]

github-actions[bot] commented on issue #20461:
URL: https://github.com/apache/airflow/issues/20461#issuecomment-1134028387

   This issue has been automatically marked as stale because it has been open 
for 30 days with no response from the author. It will be closed in next 7 days 
if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #21106: Fix StatD timing metric units

[GitBox]

github-actions[bot] commented on PR #21106:
URL: https://github.com/apache/airflow/pull/21106#issuecomment-1134028380

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #22666: Cli next execution bug

[GitBox]

github-actions[bot] closed pull request #22666: Cli next execution bug
URL: https://github.com/apache/airflow/pull/22666


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] erdos2n commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

erdos2n commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1134019825

   @uranusjr I agree, the first syntax best aligns with the current 
implementation of the dynamic mapping methods. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

uranusjr commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1134017810

   Your original syntax is correct. 
(https://github.com/apache/airflow/issues/23833#issuecomment-1133900501 is 
wrong.) This looks like a bug in the serialisation code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #21942: Improve ElasticsearchTaskHandler

[GitBox]

potiuk commented on PR #21942:
URL: https://github.com/apache/airflow/pull/21942#issuecomment-1134008649

   Looks like static checks/docs failing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23641: fix: DatabricksSubmitRunOperator and DatabricksRunNowOperator cannot define .json as template_ext (#23622)

[GitBox]

potiuk commented on PR #23641:
URL: https://github.com/apache/airflow/pull/23641#issuecomment-1134008325

   Tests are failing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #19596: MsSqlHook: implement _generate_insert_sql

[GitBox]

potiuk commented on PR #19596:
URL: https://github.com/apache/airflow/pull/19596#issuecomment-1134007643

   I rebased it - let's see. @dstandish ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23859: update K8S-KIND to 0.14.0

[GitBox]

github-actions[bot] commented on PR #23859:
URL: https://github.com/apache/airflow/pull/23859#issuecomment-1134007245

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1134006492

   Tested all change I've been involved in. All good :).
   
   The most important #23723 (passing deprecated parameters to "run_as_user" - 
works as expected).
   ```
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:141} INFO - Running on host: 
01b98cdf7c4d
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:142} INFO - Running: 
['sudo', '-E', '-H', '-u', '***', '***', 'tasks', 'run', 
'example_bash_operator', 'runme_0', 'manual__2022-05-22T22:40:11.200418+00:00', 
'--job-id', '44', '--raw', '--subdir', 
'/usr/local/lib/python3.8/site-packages/***/example_dags/example_bash_operator.py',
 '--cfg-path', '/tmp/tmpqmbfe35d', '--error-file', '/tmp/tmpvqn56f5f']
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:123} INFO - Job 44: Subtask 
runme_0 /usr/local/lib/python3.8/site-packages/***/configuration.py:525: 
DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the 
sql_alchemy_conn option in [database] - the old setting has been used, but 
please update your config.
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:123} INFO - Job 44: Subtask 
runme_0   option = self._get_environment_variables(deprecated_key, 
deprecated_section, key, section)
   ```
   
   Killing standalone #23274 also works:
   ```
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:141} INFO - Running on host: 
01b98cdf7c4d
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:142} INFO - Running: 
['sudo', '-E', '-H', '-u', '***', '***', 'tasks', 'run', 
'example_bash_operator', 'runme_0', 'manual__2022-05-22T22:40:11.200418+00:00', 
'--job-id', '44', '--raw', '--subdir', 
'/usr/local/lib/python3.8/site-packages/***/example_dags/example_bash_operator.py',
 '--cfg-path', '/tmp/tmpqmbfe35d', '--error-file', '/tmp/tmpvqn56f5f']
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:123} INFO - Job 44: Subtask 
runme_0 /usr/local/lib/python3.8/site-packages/***/configuration.py:525: 
DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the 
sql_alchemy_conn option in [database] - the old setting has been used, but 
please update your config.
   [2022-05-22, 22:40:11 UTC] {base_task_runner.py:123} INFO - Job 44: Subtask 
runme_0   option = self._get_environment_variables(deprecated_key, 
deprecated_section, key, section)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Thrameos commented on pull request #23847: Add limit for JPype1

[GitBox]

Thrameos commented on PR #23847:
URL: https://github.com/apache/airflow/pull/23847#issuecomment-1133982152

   I think that you will need to ping marsher.  I posted source dist when I 
posted to gitlab.  It looks like he didn't copy the full release by mistake.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133971193

   All images are refreshed - I also opened 
https://github.com/apache/airflow/issues/23858 to investigate why it happened.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new issue, #23858: Main providers for Airflow downgrade cncf.kubernetes provider to 3.1.2

[GitBox]

potiuk opened a new issue, #23858:
URL: https://github.com/apache/airflow/issues/23858

   ### Apache Airflow version
   
   main (development)
   
   ### What happened
   
   The cncf.kubernetes in constraints should be bumped to 4.0.2 
(automatically), but for some reason it is downgraded to 3.1.2 - we need to 
investigate why.
   
   ### What you think should happen instead
   
   the cncf.kubernetes should be updated to 4.0.2.
   
   ### How to reproduce
   
   Just run generate constraints - all the providers are correctly set to 
latest version but only cncf.kubernetes is 3.1.2
   
   ### Operating System
   
   Any
   
   ### Versions of Apache Airflow Providers
   
   cncf.kubernetes == 3.1.2
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   CI
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133963282

   Yeah. 3.7 image already pushed and has the right cncf.kubernetes provider: 
   
   ```
   [jarek@Hyperion:~] 3s 2 % docker run -it apache/airflow:2.3.1rc1-python3.7 
providers list
   
   package_name | description   
   | version
   
=+==+
   apache-airflow-providers-amazon  | Amazon integration (including 
Amazon Web Services (AWS) https://aws.amazon.com/) | 3.4.0
   apache-airflow-providers-celery  | Celery 
http://www.celeryproject.org/   
  | 2.1.4
   apache-airflow-providers-cncf-kubernetes | Kubernetes https://kubernetes.io/ 
   | 4.0.2
   apache-airflow-providers-docker  | Docker 
https://docs.docker.com/install/
  | 2.7.0
   apache-airflow-providers-elasticsearch   | Elasticsearch 
https://www.elastic.co/elasticsearch   
| 3.0.3
   apache-airflow-providers-ftp | File Transfer Protocol (FTP) 
https://tools.ietf.org/html/rfc114  | 2.1.2
   apache-airflow-providers-google  | Google services including:
   | 7.0.0
|   
   |
|   - Google Ads 
https://ads.google.com/   |
|   - Google Cloud (GCP) 
https://cloud.google.com/ |
|   - Google Firebase 
https://firebase.google.com/ |
|   - Google LevelDB 
https://github.com/google/leveldb/|
|   - Google Marketing Platform 
https://marketingplatform.google.com/  |
|   - Google Workspace 
https://workspace.google.pl/ (formerly Google Suite)|
   apache-airflow-providers-grpc| gRPC https://grpc.io/ 
   | 2.0.4
   apache-airflow-providers-hashicorp   | Hashicorp including Hashicorp 
Vault https://www.vaultproject.io/ | 2.2.0
   apache-airflow-providers-http| Hypertext Transfer Protocol 
(HTTP) https://www.w3.org/Protocols/ | 2.1.2
   apache-airflow-providers-imap| Internet Message Access Protocol 
(IMAP) https://tools.ietf.org/html/rfc3501  | 2.2.3
   apache-airflow-providers-microsoft-azure | Microsoft Azure 
https://azure.microsoft.com/ | 
3.9.0
   apache-airflow-providers-mysql   | MySQL 
https://www.mysql.com/products/ 
   | 2.2.3
   apache-airflow-providers-odbc| ODBC 
https://github.com/mkleehammer/pyodbc/wiki  
| 2.0.4
   apache-airflow-providers-postgres| PostgreSQL 
https://www.postgresql.org/ 
  | 4.1.0
   apache-airflow-providers-redis   | Redis https://redis.io/   
   | 2.0.4
   apache-airflow-providers-sendgrid| Sendgrid https://sendgrid.com/
   | 2.0.4
   apache-airflow-providers-sftp| SSH File Transfer Protocol (SFTP) 
https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/ | 2.6.0
   apache-airflow-providers-slack   | Slack https://slack.com/  
   | 4.2.3
   apache-airflow-providers-sqlite  | SQLite https://www.sqlite.org/
   | 2.1.3
   apache-airflow-providers-ssh | Secure Shell (SSH) 
https://tools.ietf.org/html/rfc4251   | 
2.4.4
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please 

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133961327

   Thanks for spotting it @gmsantos !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133960812

   I fixed it in the constraints (and rebuilding the images).
   
   @raphaelauv -> the 
https://github.com/astronomer/astronomer-providers/issues/372 could be related 
if you installed rc1 with the constraints containing 3.1.2. 
   
   I will look in detail why the downgrade happened. It could be related to the 
rc* behaviour of `pip` (previously we bumped it manually to 4.0 because of this 
but it should not be the case). I double checked all other constraints and it 
seems that it has only happened for cncf.kubernetes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] annotated tag constraints-2.3.1rc1 updated (142d4d0e12 -> 6f7831c93c)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to annotated tag constraints-2.3.1rc1
in repository https://gitbox.apache.org/repos/asf/airflow.git


*** WARNING: tag constraints-2.3.1rc1 was modified! ***

from 142d4d0e12 (tag)
  to 6f7831c93c (tag)
 tagging d18aee02142e0ab285101c45c60de1c81ef7e7ad (commit)
 replaces constraints-latest
  by Jarek Potiuk
  on Sun May 22 21:28:46 2022 +0200

- Log -
Constraints for Apache Airflow 2.3.1rc1
-BEGIN PGP SIGNATURE-

iQGzBAABCAAdFiEE5v/YhWj+/iG6EKaNIqqgRFxnaxEFAmKKjvIACgkQIqqgRFxn
axEQeQv9H/srrkTWo2cfeomUUjI5ylF7YL8XiikRBZ9tKV/dmN695u06zgx4Aghk
R4ZCcx81vtFs22yHpeZ6bzDuVc9ONbalj59ddtT2PVe3VyNHyo6S3qZ0W/qY6lRy
3vSeW2rwpazUobV8dztLIkNllD4hV9/F3WJrYns/4DbgxqAhUEBOiTfkSZ5eTM/M
2+c611uHesGhqWzct9IpLdjv4hqHBo4dBviZdSLSnKHRnWLwcpuzaTmVdQJrnBQP
TP1iTjzZyqlWtRuYTLCsnrJzZmTuk9LgP5aB0J+BfaZjsNl2xWQ1KGQnMBwl8zJT
Lfs/a4s8tFBu85IsgoUdz4FsdWwISqBGLQ5mzq3N8LDJhWBzQY4rukLO3uVR6mNr
v+3ANH/ckj+XPdkOxL5FxFqAOc6vocxshywK5XW90fuwaHLw5LRujz64dmUFdCEM
6rZJxjIY8VkRZV4T8CkVSRDduG1Y1rpjcbbvkjw853UBgoiN9kBYxsQ/eoVJWFoy
ogITReid
=ZQwQ
-END PGP SIGNATURE-
---

from bc44527fc3 Updating constraints. Build id:2363984792
 add d18aee0214 Update cncf-kubernetes to 4.0.2 version

No new revisions were added by this update.

Summary of changes:
 .bash_aliases  | 0
 .bash_history  | 3 -
 .build/.BACKEND| 1 -
 .build/.EXECUTOR   | 1 -
 .build/.HELM_VERSION   | 1 -
 .build/.KIND_VERSION   | 1 -
 .build/.KUBERNETES_MODE| 1 -
 .build/.KUBERNETES_VERSION | 1 -
 .build/.MSSQL_VERSION  | 1 -
 .build/.MYSQL_VERSION  | 1 -
 .build/.PLATFORM   | 1 -
 .build/.POSTGRES_VERSION   | 1 -
 .build/.PYTHON_MAJOR_MINOR_VERSION | 1 -
 .build/dc_ci   |71 -
 .build/main/.built_3.7 | 0
 .../main/3.7/CI/airflow-copy-.dockerignore.md5sum  | 1 -
 .../main/3.7/CI/airflow-copy-Dockerfile.ci.md5sum  | 1 -
 .build/main/3.7/CI/airflow-copy-setup.cfg.md5sum   | 1 -
 .build/main/3.7/CI/airflow-copy-setup.py.md5sum| 1 -
 .build/main/3.7/CI/docker-common.sh.md5sum | 1 -
 .../3.7/CI/docker-compile_www_assets.sh.md5sum | 1 -
 ...ocker-install_additional_dependencies.sh.md5sum | 1 -
 .../main/3.7/CI/docker-install_airflow.sh.md5sum   | 1 -
 ..._airflow_dependencies_from_branch_tip.sh.md5sum | 1 -
 ...ker-install_from_docker_context_files.sh.md5sum | 1 -
 .build/main/3.7/CI/docker-install_mysql.sh.md5sum  | 1 -
 .../main/3.7/CI/docker-install_postgres.sh.md5sum  | 1 -
 .build/main/3.7/CI/ui-package.json.md5sum  | 1 -
 .build/main/3.7/CI/ui-yarn.lock.md5sum | 1 -
 .build/main/3.7/CI/www-package.json.md5sum | 1 -
 .build/main/3.7/CI/www-webpack.config.js.md5sum| 1 -
 .build/main/3.7/CI/www-yarn.lock.md5sum| 1 -
 .gitignore | 2 -
 .inputrc   | 0
 .mypy_cache/3.7/@plugins_snapshot.json | 1 -
 .mypy_cache/3.7/_ast.data.json | 1 -
 .mypy_cache/3.7/_ast.meta.json | 1 -
 .mypy_cache/3.7/_collections_abc.data.json | 1 -
 .mypy_cache/3.7/_collections_abc.meta.json | 1 -
 .mypy_cache/3.7/_typeshed/__init__.data.json   | 1 -
 .mypy_cache/3.7/_typeshed/__init__.meta.json   | 1 -
 .mypy_cache/3.7/abc.data.json  | 1 -
 .mypy_cache/3.7/abc.meta.json  | 1 -
 .mypy_cache/3.7/array.data.json| 1 -
 .mypy_cache/3.7/array.meta.json| 1 -
 .mypy_cache/3.7/ast.data.json  | 1 -
 .mypy_cache/3.7/ast.meta.json  | 1 -
 .mypy_cache/3.7/builtins.data.json | 1 -
 .mypy_cache/3.7/builtins.meta.json | 1 -
 .mypy_cache/3.7/codecs.data.json   | 1 -
 .mypy_cache/3.7/codecs.meta.json   | 1 -
 .mypy_cache/3.7/collections/__init__.data.json | 1 -
 .mypy_cache/3.7/collections/__init__.meta.json | 1 -
 .mypy_cache/3.7/dev/__init__.data.json | 1 -
 .mypy_cache/3.7/dev/__init__.meta.json | 1 -
 .mypy_cache/3.7/dev/breeze.data.json   | 1 -
 .mypy_cache/3.7/dev/breeze.meta.json  

[GitHub] [airflow] potiuk commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

potiuk commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133956086

   It should not be the case. Looking at it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix databricks tests (#23856)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a844565799 Fix databricks tests (#23856)
a844565799 is described below

commit a8445657996f52b3ac5ce40a535d9c397c204d36
Author: pierrejeambrun 
AuthorDate: Mon May 23 03:00:37 2022 +0800

Fix databricks tests (#23856)
---
 tests/providers/databricks/operators/test_databricks.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/providers/databricks/operators/test_databricks.py 
b/tests/providers/databricks/operators/test_databricks.py
index 895beb18e5..97551eda03 100644
--- a/tests/providers/databricks/operators/test_databricks.py
+++ b/tests/providers/databricks/operators/test_databricks.py
@@ -186,7 +186,7 @@ class TestDatabricksSubmitRunOperator(unittest.TestCase):
 'git_branch': 'main',
 }
 op = DatabricksSubmitRunOperator(task_id=TASK_ID, 
git_source=git_source, json=json)
-expected = databricks_operator._deep_string_coerce(
+expected = utils.deep_string_coerce(
 {
 'new_cluster': NEW_CLUSTER,
 'notebook_task': NOTEBOOK_TASK,

[GitHub] [airflow] potiuk merged pull request #23856: Fix Databricks Operator tests

[GitBox]

potiuk merged PR #23856:
URL: https://github.com/apache/airflow/pull/23856


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] gmcrocetti opened a new pull request, #23857: Create `get_partition` and `create_partition` methods in `GlueCatalogHook`

[GitBox]

gmcrocetti opened a new pull request, #23857:
URL: https://github.com/apache/airflow/pull/23857

Add partition related methods into GlueCatalogHook:
   * `get_partition` to retrieve a Partition;
   * `create_partition` to create a Partition.
   
   closes: #19272
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragement file, named `{pr_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on pull request #19736: Add Databricks Deferrable Operators

[GitBox]

pierrejeambrun commented on PR #19736:
URL: https://github.com/apache/airflow/pull/19736#issuecomment-1133943375

   @potiuk I think this introduced a small issue. (old variable supposed to be 
remove). Tests are failing.
   
   PR for a proposed fix -> https://github.com/apache/airflow/pull/23856


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pierrejeambrun opened a new pull request, #23856: Fix Databricks Operator tests

[GitBox]

pierrejeambrun opened a new pull request, #23856:
URL: https://github.com/apache/airflow/pull/23856

   I think there is a leftover of old code here, that make tests fail due to 
https://github.com/apache/airflow/pull/19736
   
   Best
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragement file, named `{pr_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] erdos2n commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

erdos2n commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1133936446

   @snjypl here is the code and error message when it runs in Airflow. So the 
UI Error does disappear, but it still isn't working. 
   
   ```python
   import logging
   from airflow.decorators import dag, task
   from airflow.operators.python import PythonOperator
   
   from airflow.utils.dates import datetime
   
   def log_strings(string):
   logging.info(f"here is the string = {string}")
   
   @dag(
   dag_id='dynamic_dag_test',
   schedule_interval=None,
   start_date=datetime(2021, 1, 1),
   catchup=False,
   tags=['example', 'dynamic_tasks']
   )
   def tutorial_taskflow_api_etl():
   op2 = (PythonOperator
  .partial(task_id="logging_with_operator_task",
   python_callable=log_strings)
  .expand(op_kwargs={"string":["a", "b", "c"]}))
   
   return op2
   
   
   tutorial_etl_dag = tutorial_taskflow_api_etl()
   ```
   
   **Error message:**
   ```python
   [2022-05-22, 17:06:49 UTC] {taskinstance.py:1376} INFO - Executing 
 on 2022-05-22 
17:06:48.858540+00:00
   [2022-05-22, 17:06:49 UTC] {standard_task_runner.py:52} INFO - Started 
process 290 to run task
   [2022-05-22, 17:06:49 UTC] {standard_task_runner.py:79} INFO - Running: 
['airflow', 'tasks', 'run', 'dynamic_dag_test', 'logging_with_operator_task', 
'manual__2022-05-22T17:06:48.858540+00:00', '--job-id', '41', '--raw', 
'--subdir', 'DAGS_FOLDER/dynamic_dag_test.py', '--cfg-path', 
'/tmp/tmp1yeh1bff', '--map-index', '0', '--error-file', '/tmp/tmp2mzffl7i']
   [2022-05-22, 17:06:49 UTC] {standard_task_runner.py:80} INFO - Job 41: 
Subtask logging_with_operator_task
   [2022-05-22, 17:06:49 UTC] {task_command.py:369} INFO - Running 
 on host 
5b49114612fc
   [2022-05-22, 17:06:49 UTC] {taskinstance.py:1568} INFO - Exporting the 
following env vars:
   AIRFLOW_CTX_DAG_OWNER=airflow
   AIRFLOW_CTX_DAG_ID=dynamic_dag_test
   AIRFLOW_CTX_TASK_ID=logging_with_operator_task
   AIRFLOW_CTX_EXECUTION_DATE=2022-05-22T17:06:48.858540+00:00
   AIRFLOW_CTX_TRY_NUMBER=1
   AIRFLOW_CTX_DAG_RUN_ID=manual__2022-05-22T17:06:48.858540+00:00
   [2022-05-22, 17:06:49 UTC] {taskinstance.py:1888} ERROR - Task failed with 
exception
   Traceback (most recent call last):
 File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", 
line 168, in execute
   context_merge(context, self.op_kwargs, 
templates_dict=self.templates_dict)
 File "/usr/local/lib/python3.9/site-packages/airflow/utils/context.py", 
line 256, in context_merge
   context.update(*args, **kwargs)
 File "/usr/local/lib/python3.9/_collections_abc.py", line 946, in update
   for key, value in other:
   ValueError: too many values to unpack (expected 2)
   [2022-05-22, 17:06:50 UTC] {taskinstance.py:1394} INFO - Marking task as 
FAILED. dag_id=dynamic_dag_test, task_id=logging_with_operator_task, 
map_index=0, execution_date=20220522T170648, start_date=20220522T170649, 
end_date=20220522T170650
   [2022-05-22, 17:06:50 UTC] {standard_task_runner.py:92} ERROR - Failed to 
execute job 41 for task logging_with_operator_task (too many values to unpack 
(expected 2); 290)
   [2022-05-22, 17:06:50 UTC] {local_task_job.py:156} INFO - Task exited with 
return code 1
   [2022-05-22, 17:06:50 UTC] {local_task_job.py:273} INFO - 0 downstream tasks 
scheduled from follow-on schedule check
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] alexott commented on pull request #23712: Add databricks job sensors

[GitBox]

alexott commented on PR #23712:
URL: https://github.com/apache/airflow/pull/23712#issuecomment-1133935358

   In general I see this as following. It should allow to check for following:
   
   - specific run by run ID
   - Specific job by name or ID. If not specified, latest active run. Or 
specific run around give date/time.
   - Task(s) inside give job (as above).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] josh-fell commented on a diff in pull request #23712: Add databricks job sensors

[GitBox]

josh-fell commented on code in PR #23712:
URL: https://github.com/apache/airflow/pull/23712#discussion_r878530562


##
airflow/providers/databricks/sensors/databricks.py:
##
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Databricks sensors"""
+
+from typing import TYPE_CHECKING, Any, Dict, Optional, Sequence, Union
+
+from airflow.exceptions import AirflowException
+from airflow.providers.databricks.hooks.databricks import DatabricksHook
+from airflow.sensors.base import BaseSensorOperator
+from airflow.utils.context import Context
+
+if TYPE_CHECKING:
+from airflow.sensors.base import PokeReturnValue
+
+
+class DatabricksJobRunSensor(BaseSensorOperator):
+"""
+Check for the state of a submitted Databricks job run or specific task of 
a job run.
+
+:param run_id: Id of the submitted Databricks job run or specific task of 
a job run. (templated)
+:param databricks_conn_id: Reference to the :ref:`Databricks connection 
`.
+By default and in the common case this will be ``databricks_default``. 
To use
+token based authentication, provide the key ``token`` in the extra 
field for the
+connection and create the key ``host`` and leave the ``host`` field 
empty.
+:param retry_limit: Amount of times retry if the Databricks backend is
+unreachable. Its value must be greater than or equal to 1.
+:param retry_delay_seconds: Number of seconds to wait between retries (it
+might be a floating point number).
+:param databricks_retry_args: An optional dictionary with arguments passed 
to ``tenacity.Retrying`` class.
+"""
+
+template_fields: Sequence[str] = ('run_id',)
+
+# Databricks brand color (blue) under white text
+ui_color = '#1CB1C2'
+ui_fgcolor = '#fff'
+
+def __init__(
+self,
+*,
+run_id: int,
+databricks_conn_id: str = 'databricks_default',
+retry_limit: int = 3,
+retry_delay_seconds: int = 1,
+databricks_retry_args: Optional[Dict[Any, Any]] = None,

Review Comment:
   ```suggestion
   retry_args: Optional[Dict[Any, Any]] = None,
   ```
   This shouldn't have any underlying collision problems and would be 
consistent with the other parameter names dropping the "databricks" prefix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] alexott commented on a diff in pull request #23712: Add databricks job sensors

[GitBox]

alexott commented on code in PR #23712:
URL: https://github.com/apache/airflow/pull/23712#discussion_r878899065


##
airflow/providers/databricks/sensors/databricks.py:
##
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Databricks sensors"""
+
+from typing import TYPE_CHECKING, Any, Dict, Optional, Sequence, Union
+
+from airflow.exceptions import AirflowException
+from airflow.providers.databricks.hooks.databricks import DatabricksHook
+from airflow.sensors.base import BaseSensorOperator
+from airflow.utils.context import Context
+
+if TYPE_CHECKING:
+from airflow.sensors.base import PokeReturnValue
+
+
+class DatabricksJobRunSensor(BaseSensorOperator):
+"""
+Check for the state of a submitted Databricks job run or specific task of 
a job run.
+
+:param run_id: Id of the submitted Databricks job run or specific task of 
a job run. (templated)

Review Comment:
   Job names aren’t distinct, but it could be still handled as its done in 
RunNow operator…
   
   task names inside job are distinct 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] gmsantos commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

gmsantos commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133933602

   I see that the apache-airflow-providers-cncf-kubernetes was downgraded to 
3.1.2 in 2.3.1rc1 constraints file (it was 4.0.1 in 2.3.0 constraints).
   
   Is it not good yet to go 4.0 for this provider?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jplauri commented on pull request #19596: MsSqlHook: implement _generate_insert_sql

[GitBox]

jplauri commented on PR #19596:
URL: https://github.com/apache/airflow/pull/19596#issuecomment-1133920733

   > Please add a test for this
   
   Should this be good to go now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] snjypl commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

snjypl commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1133913518

   yes @erdos2n  i was able to get that work. can you please share the error 
you are getting? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #23849: Fix Amazon EKS example DAG raises warning during Imports

[GitBox]

boring-cyborg[bot] commented on PR #23849:
URL: https://github.com/apache/airflow/pull/23849#issuecomment-1133908387

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #23849: Fix Amazon EKS example DAG raises warning during Imports

[GitBox]

potiuk merged PR #23849:
URL: https://github.com/apache/airflow/pull/23849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix Amazon EKS example DAG raises warning during Imports (#23849)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 5d2296becb Fix Amazon EKS example DAG raises warning during Imports 
(#23849)
5d2296becb is described below

commit 5d2296becb9401df6ca58bb7d15d6655eb168aed
Author: Vedant Bhamare <55763604+dark-knigh...@users.noreply.github.com>
AuthorDate: Sun May 22 19:55:20 2022 +0530

Fix Amazon EKS example DAG raises warning during Imports (#23849)


Co-authored-by: eladkal <45845474+elad...@users.noreply.github.com>
---
 airflow/providers/amazon/aws/operators/eks.py | 20 ++--
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/airflow/providers/amazon/aws/operators/eks.py 
b/airflow/providers/amazon/aws/operators/eks.py
index ef40cab2e1..600c4d3aa1 100644
--- a/airflow/providers/amazon/aws/operators/eks.py
+++ b/airflow/providers/amazon/aws/operators/eks.py
@@ -275,23 +275,23 @@ class EksCreateNodegroupOperator(BaseOperator):
 self.create_nodegroup_kwargs = create_nodegroup_kwargs or {}
 self.aws_conn_id = aws_conn_id
 self.region = region
-nodegroup_subnets_list: List[str] = []
-if isinstance(nodegroup_subnets, str):
-if nodegroup_subnets != "":
+self.nodegroup_subnets = nodegroup_subnets
+super().__init__(**kwargs)
+
+def execute(self, context: 'Context'):
+if isinstance(self.nodegroup_subnets, str):
+nodegroup_subnets_list: List[str] = []
+if self.nodegroup_subnets != "":
 try:
-nodegroup_subnets_list = cast(List, 
literal_eval(nodegroup_subnets))
+nodegroup_subnets_list = cast(List, 
literal_eval(self.nodegroup_subnets))
 except ValueError:
 self.log.warning(
 "The nodegroup_subnets should be List or string 
representing "
 "Python list and is %s. Defaulting to []",
-nodegroup_subnets,
+self.nodegroup_subnets,
 )
-else:
-nodegroup_subnets_list = nodegroup_subnets
-self.nodegroup_subnets = nodegroup_subnets_list
-super().__init__(**kwargs)
+self.nodegroup_subnets = nodegroup_subnets_list
 
-def execute(self, context: 'Context'):
 eks_hook = EksHook(
 aws_conn_id=self.aws_conn_id,
 region_name=self.region,

[GitHub] [airflow] potiuk closed issue #23756: Amazon EKS example DAG raises warnning

[GitBox]

potiuk closed issue #23756: Amazon EKS example DAG raises warnning
URL: https://github.com/apache/airflow/issues/23756


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add Deferrable Databricks operators (#19736)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 6150d28323 Add Deferrable Databricks operators (#19736)
6150d28323 is described below

commit 6150d283234b48f86362fd4da856e282dd91ebb4
Author: Eugene Karimov <13220923+eskari...@users.noreply.github.com>
AuthorDate: Sun May 22 16:22:49 2022 +0200

Add Deferrable Databricks operators (#19736)
---
 airflow/providers/databricks/hooks/databricks.py   |  29 ++
 .../providers/databricks/hooks/databricks_base.py  | 237 +-
 .../providers/databricks/operators/databricks.py   | 106 +--
 airflow/providers/databricks/triggers/__init__.py  |  17 +
 .../providers/databricks/triggers/databricks.py|  77 +
 airflow/providers/databricks/utils/__init__.py |  16 +
 airflow/providers/databricks/utils/databricks.py   |  69 
 .../operators/run_now.rst  |   7 +
 .../operators/submit_run.rst   |   7 +
 setup.py   |   2 +
 .../providers/databricks/hooks/test_databricks.py  | 352 -
 .../databricks/operators/test_databricks.py| 239 +++---
 tests/providers/databricks/triggers/__init__.py|  17 +
 .../databricks/triggers/test_databricks.py | 153 +
 tests/providers/databricks/utils/__init__.py   |  16 +
 tests/providers/databricks/utils/databricks.py |  62 
 16 files changed, 1313 insertions(+), 93 deletions(-)

diff --git a/airflow/providers/databricks/hooks/databricks.py 
b/airflow/providers/databricks/hooks/databricks.py
index 7911660412..400bbe8955 100644
--- a/airflow/providers/databricks/hooks/databricks.py
+++ b/airflow/providers/databricks/hooks/databricks.py
@@ -25,6 +25,7 @@ operators talk to the
 or the ``api/2.1/jobs/runs/submit``
 `endpoint 
`_.
 """
+import json
 from typing import Any, Dict, List, Optional
 
 from requests import exceptions as requests_exceptions
@@ -92,6 +93,13 @@ class RunState:
 def __repr__(self) -> str:
 return str(self.__dict__)
 
+def to_json(self) -> str:
+return json.dumps(self.__dict__)
+
+@classmethod
+def from_json(cls, data: str) -> 'RunState':
+return RunState(**json.loads(data))
+
 
 class DatabricksHook(BaseDatabricksHook):
 """
@@ -198,6 +206,16 @@ class DatabricksHook(BaseDatabricksHook):
 response = self._do_api_call(GET_RUN_ENDPOINT, json)
 return response['run_page_url']
 
+async def a_get_run_page_url(self, run_id: int) -> str:
+"""
+Async version of `get_run_page_url()`.
+:param run_id: id of the run
+:return: URL of the run page
+"""
+json = {'run_id': run_id}
+response = await self._a_do_api_call(GET_RUN_ENDPOINT, json)
+return response['run_page_url']
+
 def get_job_id(self, run_id: int) -> int:
 """
 Retrieves job_id from run_id.
@@ -229,6 +247,17 @@ class DatabricksHook(BaseDatabricksHook):
 state = response['state']
 return RunState(**state)
 
+async def a_get_run_state(self, run_id: int) -> RunState:
+"""
+Async version of `get_run_state()`.
+:param run_id: id of the run
+:return: state of the run
+"""
+json = {'run_id': run_id}
+response = await self._a_do_api_call(GET_RUN_ENDPOINT, json)
+state = response['state']
+return RunState(**state)
+
 def get_run_state_str(self, run_id: int) -> str:
 """
 Return the string representation of RunState.
diff --git a/airflow/providers/databricks/hooks/databricks_base.py 
b/airflow/providers/databricks/hooks/databricks_base.py
index 6e0f1b44d8..5b18dad930 100644
--- a/airflow/providers/databricks/hooks/databricks_base.py
+++ b/airflow/providers/databricks/hooks/databricks_base.py
@@ -28,11 +28,19 @@ import time
 from typing import Any, Dict, Optional, Tuple
 from urllib.parse import urlparse
 
+import aiohttp
 import requests
 from requests import PreparedRequest, exceptions as requests_exceptions
 from requests.auth import AuthBase, HTTPBasicAuth
 from requests.exceptions import JSONDecodeError
-from tenacity import RetryError, Retrying, retry_if_exception, 
stop_after_attempt, wait_exponential
+from tenacity import (
+AsyncRetrying,
+RetryError,
+Retrying,
+retry_if_exception,
+stop_after_attempt,
+wait_exponential,
+)
 
 from airflow import __version__
 from airflow.exceptions import AirflowException
@@ -135,6 +143,14 @@ class BaseDatabricksHook(BaseHook):
 
 return host
 
+async def __aenter__(self):
+self._session = aiohttp.ClientSession()
+return self
+
+async def __aexit__(self, *err):
+await 

[GitHub] [airflow] potiuk merged pull request #19736: Add Databricks Deferrable Operators

[GitBox]

potiuk merged PR #19736:
URL: https://github.com/apache/airflow/pull/19736


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #18999: Update databricks provider to use TriggerOperator

[GitBox]

potiuk closed issue #18999: Update databricks provider to use TriggerOperator
URL: https://github.com/apache/airflow/issues/18999


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #19736: Add Databricks Deferrable Operators

[GitBox]

potiuk commented on PR #19736:
URL: https://github.com/apache/airflow/pull/19736#issuecomment-1133907811

   LGTM. We can remove duplication later I think :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

[GitBox]

pierrejeambrun commented on issue #23852:
URL: https://github.com/apache/airflow/issues/23852#issuecomment-1133903495

   Hello Guys,
   
   https://github.com/apache/airflow/pull/23392 is working fine :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] erdos2n commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

erdos2n commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1133903250

   @snjypl I tried this and it still didn't work, were you able to get it to 
work? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (65f3b18fc1 -> e54ca47262)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 65f3b18fc1 Sql to gcs with exclude columns (#23695)
 add e54ca47262 Add support for associating  custom tags to job runs 
submitted via EmrContainerOperator (#23769)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/amazon/aws/hooks/emr.py   | 3 +++
 airflow/providers/amazon/aws/operators/emr.py   | 5 +
 tests/providers/amazon/aws/operators/test_emr_containers.py | 3 ++-
 3 files changed, 10 insertions(+), 1 deletion(-)

[GitHub] [airflow] potiuk merged pull request #23769: Add support for associating custom tags to job runs submitted via EmrContainerOperator

[GitBox]

potiuk merged PR #23769:
URL: https://github.com/apache/airflow/pull/23769


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #23765: Add support of tags assignment to job runs in EmrContainerOperator

[GitBox]

potiuk closed issue #23765: Add support of tags assignment to job runs in 
EmrContainerOperator 
URL: https://github.com/apache/airflow/issues/23765


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on pull request #23678: Highlight task states by hovering on legend row

[GitBox]

pierrejeambrun commented on PR #23678:
URL: https://github.com/apache/airflow/pull/23678#issuecomment-1133902509

   Everything is working now, thank you @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] snjypl commented on issue #23833: Dynamic Task Mapping not working with op_kwargs in PythonOperator

[GitBox]

snjypl commented on issue #23833:
URL: https://github.com/apache/airflow/issues/23833#issuecomment-1133900501

   @erdos2n  i guess, you need to pass `op_kwargs` as a `dict` : 
   ```
 .expand(op_kwargs={"string":[ "a", "b","c"]}))
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23848: The first system test for slack (implemented according to AIP-47)

[GitBox]

potiuk commented on PR #23848:
URL: https://github.com/apache/airflow/pull/23848#issuecomment-1133897561

   Static checks fail - Recommend pre-commit installation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #19736: Add Databricks Deferrable Operators

[GitBox]

github-actions[bot] commented on PR #19736:
URL: https://github.com/apache/airflow/pull/19736#issuecomment-1133896588

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23848: The first system test for slack (implemented according to AIP-47)

[GitBox]

potiuk commented on PR #23848:
URL: https://github.com/apache/airflow/pull/23848#issuecomment-1133891086

   Just small nit - I went ahead and added a fixup :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #23848: The first system test for slack (implemented according to AIP-47)

[GitBox]

potiuk commented on code in PR #23848:
URL: https://github.com/apache/airflow/pull/23848#discussion_r878857911


##
tests/system/providers/slack/example_slack.py:
##
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Google BigQuery service testing dataset operations.

Review Comment:
   ```suggestion
   Example Airflow DAG for SlackAPIFileOperator.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Sql to gcs with exclude columns (#23695)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 65f3b18fc1 Sql to gcs with exclude columns (#23695)
65f3b18fc1 is described below

commit 65f3b18fc1142c0d23e715fa1a98f21662df9584
Author: 서재권(Data Platform) <90180644+jaegwon...@users.noreply.github.com>
AuthorDate: Sun May 22 21:50:49 2022 +0900

Sql to gcs with exclude columns (#23695)
---
 .../providers/google/cloud/transfers/sql_to_gcs.py |  16 ++--
 tests/providers/google/cloud/transfers/temp-file   | Bin 3739 -> 0 bytes
 .../google/cloud/transfers/test_sql_to_gcs.py  |  29 +
 3 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/airflow/providers/google/cloud/transfers/sql_to_gcs.py 
b/airflow/providers/google/cloud/transfers/sql_to_gcs.py
index 0df46bc777..46e1ad505d 100644
--- a/airflow/providers/google/cloud/transfers/sql_to_gcs.py
+++ b/airflow/providers/google/cloud/transfers/sql_to_gcs.py
@@ -71,6 +71,7 @@ class BaseSQLToGCSOperator(BaseOperator):
 If set as a sequence, the identities from the list must grant
 Service Account Token Creator IAM role to the directly preceding 
identity, with first
 account from the list granting this role to the originating account 
(templated).
+:param exclude_columns: set of columns to exclude from transmission
 """
 
 template_fields: Sequence[str] = (
@@ -103,9 +104,13 @@ class BaseSQLToGCSOperator(BaseOperator):
 gcp_conn_id: str = 'google_cloud_default',
 delegate_to: Optional[str] = None,
 impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+exclude_columns=None,
 **kwargs,
 ) -> None:
 super().__init__(**kwargs)
+if exclude_columns is None:
+exclude_columns = set()
+
 self.sql = sql
 self.bucket = bucket
 self.filename = filename
@@ -120,6 +125,7 @@ class BaseSQLToGCSOperator(BaseOperator):
 self.gcp_conn_id = gcp_conn_id
 self.delegate_to = delegate_to
 self.impersonation_chain = impersonation_chain
+self.exclude_columns = exclude_columns
 
 def execute(self, context: 'Context'):
 self.log.info("Executing query")
@@ -165,7 +171,9 @@ class BaseSQLToGCSOperator(BaseOperator):
 names in GCS, and values are file handles to local files that
 contain the data for the GCS objects.
 """
-schema = list(map(lambda schema_tuple: schema_tuple[0], 
cursor.description))
+org_schema = list(map(lambda schema_tuple: schema_tuple[0], 
cursor.description))
+schema = [column for column in org_schema if column not in 
self.exclude_columns]
+
 col_type_dict = self._get_col_type_dict()
 file_no = 0
 
@@ -314,7 +322,11 @@ class BaseSQLToGCSOperator(BaseOperator):
 schema = self.schema
 else:
 self.log.info("Starts generating schema")
-schema = [self.field_to_bigquery(field) for field in 
cursor.description]
+schema = [
+self.field_to_bigquery(field)
+for field in cursor.description
+if field[0] not in self.exclude_columns
+]
 
 if isinstance(schema, list):
 schema = json.dumps(schema, sort_keys=True)
diff --git a/tests/providers/google/cloud/transfers/temp-file 
b/tests/providers/google/cloud/transfers/temp-file
deleted file mode 100644
index d2282fc46c..00
Binary files a/tests/providers/google/cloud/transfers/temp-file and /dev/null 
differ
diff --git a/tests/providers/google/cloud/transfers/test_sql_to_gcs.py 
b/tests/providers/google/cloud/transfers/test_sql_to_gcs.py
index 525e04bd0e..824ab8ff31 100644
--- a/tests/providers/google/cloud/transfers/test_sql_to_gcs.py
+++ b/tests/providers/google/cloud/transfers/test_sql_to_gcs.py
@@ -61,6 +61,12 @@ APP_JSON = "application/json"
 
 OUTPUT_DF = pd.DataFrame([['convert_type_return_value'] * 3] * 3, 
columns=COLUMNS)
 
+EXCLUDE_COLUMNS = set('column_c')
+NEW_COLUMNS = [c for c in COLUMNS if c not in EXCLUDE_COLUMNS]
+OUTPUT_DF_WITH_EXCLUDE_COLUMNS = pd.DataFrame(
+[['convert_type_return_value'] * len(NEW_COLUMNS)] * 3, columns=NEW_COLUMNS
+)
+
 
 class DummySQLToGCSOperator(BaseSQLToGCSOperator):
 def field_to_bigquery(self, field) -> Dict[str, str]:
@@ -287,3 +293,26 @@ class TestBaseSQLToGCSOperator(unittest.TestCase):
 file.flush()
 df = pd.read_parquet(file.name)
 assert df.equals(OUTPUT_DF)
+
+def test__write_local_data_files_json_with_exclude_columns(self):
+op = DummySQLToGCSOperator(
+sql=SQL,
+bucket=BUCKET,
+filename=FILENAME,
+task_id=TASK_ID,
+schema_filename=SCHEMA_FILE,
+export_format="json",
+

[GitHub] [airflow] potiuk merged pull request #23695: Sql to gcs with exclude columns

[GitBox]

potiuk merged PR #23695:
URL: https://github.com/apache/airflow/pull/23695


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #23695: Sql to gcs with exclude columns

[GitBox]

boring-cyborg[bot] commented on PR #23695:
URL: https://github.com/apache/airflow/pull/23695#issuecomment-1133889833

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23678: Highlight task states by hovering on legend row

[GitBox]

potiuk commented on PR #23678:
URL: https://github.com/apache/airflow/pull/23678#issuecomment-113316

   Hopefully it was that one that caused it: 
https://github.com/apache/airflow/pull/23785


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23678: Highlight task states by hovering on legend row

[GitBox]

potiuk commented on PR #23678:
URL: https://github.com/apache/airflow/pull/23678#issuecomment-1133888018

   @pierrejeambrun -  I could not see any reason why the build would fail.  - I 
rebased the PR hoping that this was some weird temporary issue. If it fails 
again, it measn that yarn command fails - this is something you can easily test 
locally by running the compile script in breeze (as suggested when you enter 
breeze).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23826: BigQueryInsertJobOperator is broken on any type of job except `query`

[GitBox]

potiuk commented on issue #23826:
URL: https://github.com/apache/airflow/issues/23826#issuecomment-1133885839

   Ah OK. That was not clear @wojsamjan -> can you please take a look and fix 
it ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] raphaelauv commented on issue #23826: BigQueryInsertJobOperator is broken on any type of job except `query`

[GitBox]

raphaelauv commented on issue #23826:
URL: https://github.com/apache/airflow/issues/23826#issuecomment-1133885492

   No It's the reason of the issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add number of node params only for single-node cluster in RedshiftCreateClusterOperator (#23839)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 69f444f87a Add number of node params only for single-node cluster in 
RedshiftCreateClusterOperator (#23839)
69f444f87a is described below

commit 69f444f87acef511da8f44ebd04ee435d10b7e5c
Author: pankajastro <98807258+pankajas...@users.noreply.github.com>
AuthorDate: Sun May 22 17:59:27 2022 +0530

Add number of node params only for single-node cluster in 
RedshiftCreateClusterOperator (#23839)
---
 .../amazon/aws/operators/redshift_cluster.py   |  4 +--
 .../amazon/aws/operators/test_redshift_cluster.py  | 33 --
 2 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/airflow/providers/amazon/aws/operators/redshift_cluster.py 
b/airflow/providers/amazon/aws/operators/redshift_cluster.py
index e9b1f23b6d..340e9577ef 100644
--- a/airflow/providers/amazon/aws/operators/redshift_cluster.py
+++ b/airflow/providers/amazon/aws/operators/redshift_cluster.py
@@ -176,8 +176,8 @@ class RedshiftCreateClusterOperator(BaseOperator):
 params["DBName"] = self.db_name
 if self.cluster_type:
 params["ClusterType"] = self.cluster_type
-if self.number_of_nodes:
-params["NumberOfNodes"] = self.number_of_nodes
+if self.cluster_type == "multi-node":
+params["NumberOfNodes"] = self.number_of_nodes
 if self.cluster_security_groups:
 params["ClusterSecurityGroups"] = self.cluster_security_groups
 if self.vpc_security_group_ids:
diff --git a/tests/providers/amazon/aws/operators/test_redshift_cluster.py 
b/tests/providers/amazon/aws/operators/test_redshift_cluster.py
index e29b397211..2ec6897687 100644
--- a/tests/providers/amazon/aws/operators/test_redshift_cluster.py
+++ b/tests/providers/amazon/aws/operators/test_redshift_cluster.py
@@ -41,7 +41,7 @@ class TestRedshiftCreateClusterOperator:
 assert redshift_operator.master_user_password == "Test123$"
 
 
@mock.patch("airflow.providers.amazon.aws.hooks.redshift.RedshiftHook.get_conn")
-def test_create_cluster(self, mock_get_conn):
+def test_create_single_node_cluster(self, mock_get_conn):
 redshift_operator = RedshiftCreateClusterOperator(
 task_id="task_test",
 cluster_identifier="test-cluster",
@@ -54,7 +54,36 @@ class TestRedshiftCreateClusterOperator:
 params = {
 "DBName": "dev",
 "ClusterType": "single-node",
-"NumberOfNodes": 1,
+"AutomatedSnapshotRetentionPeriod": 1,
+"ClusterVersion": "1.0",
+"AllowVersionUpgrade": True,
+"PubliclyAccessible": True,
+"Port": 5439,
+}
+mock_get_conn.return_value.create_cluster.assert_called_once_with(
+ClusterIdentifier='test-cluster',
+NodeType="dc2.large",
+MasterUsername="adminuser",
+MasterUserPassword="Test123$",
+**params,
+)
+
+
@mock.patch("airflow.providers.amazon.aws.hooks.redshift.RedshiftHook.get_conn")
+def test_create_multi_node_cluster(self, mock_get_conn):
+redshift_operator = RedshiftCreateClusterOperator(
+task_id="task_test",
+cluster_identifier="test-cluster",
+node_type="dc2.large",
+number_of_nodes=3,
+master_username="adminuser",
+master_user_password="Test123$",
+cluster_type="multi-node",
+)
+redshift_operator.execute(None)
+params = {
+"DBName": "dev",
+"ClusterType": "multi-node",
+"NumberOfNodes": 3,
 "AutomatedSnapshotRetentionPeriod": 1,
 "ClusterVersion": "1.0",
 "AllowVersionUpgrade": True,

[GitHub] [airflow] potiuk merged pull request #23839: Fix ``RedshiftCreateClusterOperator`` for single-node cluster type

[GitBox]

potiuk merged PR #23839:
URL: https://github.com/apache/airflow/pull/23839


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix UnboundLocalError when sql is empty list in DatabricksSqlHook (#23815)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new cf5a78e91c Fix UnboundLocalError when sql is empty list in 
DatabricksSqlHook (#23815)
cf5a78e91c is described below

commit cf5a78e91cb920e7014b76914956681aeb44b29f
Author: Dmytro Kazanzhy 
AuthorDate: Sun May 22 15:27:28 2022 +0300

Fix UnboundLocalError when sql is empty list in DatabricksSqlHook (#23815)
---
 airflow/providers/databricks/hooks/databricks_sql.py| 6 +-
 tests/providers/databricks/hooks/test_databricks_sql.py | 8 
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/airflow/providers/databricks/hooks/databricks_sql.py 
b/airflow/providers/databricks/hooks/databricks_sql.py
index afc165ee20..9d86b4dbe3 100644
--- a/airflow/providers/databricks/hooks/databricks_sql.py
+++ b/airflow/providers/databricks/hooks/databricks_sql.py
@@ -167,7 +167,11 @@ class DatabricksSqlHook(BaseDatabricksHook, DbApiHook):
 """
 if isinstance(sql, str):
 sql = self.maybe_split_sql_string(sql)
-self.log.debug("Executing %d statements", len(sql))
+
+if sql:
+self.log.debug("Executing %d statements", len(sql))
+else:
+raise ValueError("List of SQL statements is empty")
 
 conn = None
 for sql_statement in sql:
diff --git a/tests/providers/databricks/hooks/test_databricks_sql.py 
b/tests/providers/databricks/hooks/test_databricks_sql.py
index 369aa045eb..05952beabd 100644
--- a/tests/providers/databricks/hooks/test_databricks_sql.py
+++ b/tests/providers/databricks/hooks/test_databricks_sql.py
@@ -20,6 +20,8 @@
 import unittest
 from unittest import mock
 
+import pytest
+
 from airflow.models import Connection
 from airflow.providers.databricks.hooks.databricks_sql import DatabricksSqlHook
 from airflow.utils.session import provide_session
@@ -82,3 +84,9 @@ class TestDatabricksSqlHookQueryByName(unittest.TestCase):
 
 cur.execute.assert_has_calls([mock.call(q) for q in [query]])
 cur.close.assert_called()
+
+def test_no_query(self):
+for empty_statement in ([], '', '\n'):
+with pytest.raises(ValueError) as err:
+self.hook.run(sql=empty_statement)
+assert err.value.args[0] == "List of SQL statements is empty"

[GitHub] [airflow] potiuk merged pull request #23815: Fix UnboundLocalError when sql is empty list in DatabricksSqlHook

[GitBox]

potiuk merged PR #23815:
URL: https://github.com/apache/airflow/pull/23815


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix UnboundLocalError when sql is empty list in DbApiHook (#23816)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 4b5a101d29 Fix UnboundLocalError when sql is empty list in DbApiHook 
(#23816)
4b5a101d29 is described below

commit 4b5a101d29d8b0188d41abc5d079b0d363902de1
Author: Dmytro Kazanzhy 
AuthorDate: Sun May 22 15:26:57 2022 +0300

Fix UnboundLocalError when sql is empty list in DbApiHook (#23816)
---
 airflow/hooks/dbapi.py| 5 +
 tests/hooks/test_dbapi.py | 5 +
 2 files changed, 10 insertions(+)

diff --git a/airflow/hooks/dbapi.py b/airflow/hooks/dbapi.py
index 1f898706a5..da33bacca8 100644
--- a/airflow/hooks/dbapi.py
+++ b/airflow/hooks/dbapi.py
@@ -178,6 +178,11 @@ class DbApiHook(BaseHook):
 if scalar:
 sql = [sql]
 
+if sql:
+self.log.debug("Executing %d statements", len(sql))
+else:
+raise ValueError("List of SQL statements is empty")
+
 with closing(self.get_conn()) as conn:
 if self.supports_autocommit:
 self.set_autocommit(conn, autocommit)
diff --git a/tests/hooks/test_dbapi.py b/tests/hooks/test_dbapi.py
index 81a63de441..fd2bbd9132 100644
--- a/tests/hooks/test_dbapi.py
+++ b/tests/hooks/test_dbapi.py
@@ -273,3 +273,8 @@ class TestDbApiHook(unittest.TestCase):
 assert called == 2
 assert self.conn.commit.called
 assert result == [obj, obj]
+
+def test_run_no_queries(self):
+with pytest.raises(ValueError) as err:
+self.db_hook.run(sql=[])
+assert err.value.args[0] == "List of SQL statements is empty"

[GitHub] [airflow] potiuk merged pull request #23816: Fix UnboundLocalError when sql is empty list in DbApiHook

[GitBox]

potiuk merged PR #23816:
URL: https://github.com/apache/airflow/pull/23816


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23832: AWS Glue Hook to implement enhanced logging

[GitBox]

potiuk commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1133884677

   @ak-arun - maybe you would like to add Pull Request fixing it ? Seems ike an 
easy thing to do that would not require deep knowledge of Airlow itself and you 
could test it locally with your setup ? Can I assign it to you? This is the 
easiest and fastest way to get it implemented :). Otherwise I mark it as 
"good-first-issue".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23760: OOME when using GCSObjectsWithPrefixExistenceSensor on bucket with many objects

[GitBox]

potiuk commented on issue #23760:
URL: https://github.com/apache/airflow/issues/23760#issuecomment-1133884373

   Either of you @jonathan-ostrander @gpoulin can fix it :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

[GitBox]

potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1133883891

   Would love to get some feedback from our Helm people :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

[GitBox]

github-actions[bot] commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1133883753

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23826: BigQueryInsertJobOperator is broken on any type of job except `query`

[GitBox]

potiuk commented on issue #23826:
URL: https://github.com/apache/airflow/issues/23826#issuecomment-1133883527

   Do you think #23165  fixes this issue ? Can we close it @raphaelauv ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add better feedback to Breeze users about expected action timing (#23827)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new fe91db7525 Add better feedback to Breeze users about expected action 
timing (#23827)
fe91db7525 is described below

commit fe91db752525717cf5b8017f16e4bc60c6a2b26d
Author: Jarek Potiuk 
AuthorDate: Sun May 22 14:15:24 2022 +0200

Add better feedback to Breeze users about expected action timing (#23827)

There are a few actions in Breeze that might take more or less time
when invoked. This is mostly when you need to upgrade Breeze or
update to latest version of the image because some dependedncies
were added or image was modified.

While we have improved significantly the waiting time involved
now (and caching problems have been fixed to make it as fast
possible), there are still a few situations that you need to have
a good connectivity and a little time to run the upgrade. Which
is often not something you would like to loose your time on in
a number of cases when you need to do things fast.

Usually Breeeze does not force the user to perform such long
actions - it allows to continue without doing them (either by
timeout or by letting user answer "no" to question asked.

Previously Breeze have not informed the user about the exepcted
time of running such operation, but with this change it tells
what is the expected delay - thus allowing the user to make
informed action whether they want to run the upgrade or not.
---
 .../src/airflow_breeze/commands/ci_image_commands.py  | 19 +--
 dev/breeze/src/airflow_breeze/utils/reinstall.py  |  9 ++---
 2 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py 
b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index 1329ba607c..020cd30698 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -383,17 +383,21 @@ def should_we_run_the_build(build_ci_params: 
BuildCiParams, verbose: bool) -> bo
 return False
 try:
 answer = user_confirm(
-message="Do you want to build the image?", 
timeout=STANDARD_TIMEOUT, default_answer=Answer.NO
+message="Do you want to build the image (this works best when you 
have good connection and "
+"can take usually from 20 seconds to few minutes depending how old 
your image is)?",
+timeout=STANDARD_TIMEOUT,
+default_answer=Answer.NO,
 )
 if answer == answer.YES:
 if is_repo_rebased(build_ci_params.github_repository, 
build_ci_params.airflow_branch):
 return True
 else:
 get_console().print(
-"\n[warning]This might take a lot of time, we think you 
should rebase first.[/]\n"
+"\n[warning]This might take a lot of time (more than 10 
minutes) even if you have"
+"a good network connection. We think you should attempt to 
rebase first.[/]\n"
 )
 answer = user_confirm(
-"But if you really, really want - you can do it. Are you 
really sure?",
+"But if you really, really want - you can attempt it. Are 
you really sure?",
 timeout=STANDARD_TIMEOUT,
 default_answer=Answer.NO,
 )
@@ -401,8 +405,8 @@ def should_we_run_the_build(build_ci_params: BuildCiParams, 
verbose: bool) -> bo
 return True
 else:
 get_console().print(
-"[info]Please rebase your code before continuing.[/]\n"
-"Check this link to know more "
+f"[info]Please rebase your code to latest 
{build_ci_params.airflow_branch} "
+"before continuing.[/]\nCheck this link to find out 
how "
 
"https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#id15\n;
 )
 get_console().print('[error]Exiting the process[/]\n')
@@ -538,6 +542,9 @@ def rebuild_ci_image_if_needed(
 if verbose:
 get_console().print(f'[info]{build_params.image_type} image 
already built locally.[/]')
 else:
-get_console().print(f'[warning]{build_params.image_type} image not 
built locally. Forcing build.[/]')
+get_console().print(
+f'[warning]{build_params.image_type} image was never built locally 
or deleted. '
+'Forcing build.[/]'
+)
 ci_image_params.force_build = True
 build_ci_image(verbose, dry_run=dry_run, ci_image_params=ci_image_params)
diff --git 

[GitHub] [airflow] potiuk merged pull request #23827: Add better feedback to Breeze users about expected action timing

[GitBox]

potiuk merged PR #23827:
URL: https://github.com/apache/airflow/pull/23827


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23827: Add better feedback to Breeze users about expected action timing

[GitBox]

github-actions[bot] commented on PR #23827:
URL: https://github.com/apache/airflow/pull/23827#issuecomment-1133882944

   The PR is likely ready to be merged. No tests are needed as no important 
environment files, nor python files were modified by it. However, committers 
might decide that full test matrix is needed and add the 'full tests needed' 
label. Then you should rebase it to the latest main or amend the last commit of 
the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Upgrade `pip` to 22.1.1 version (just released) (#23854)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a7425dc62e Upgrade `pip` to 22.1.1 version (just released) (#23854)
a7425dc62e is described below

commit a7425dc62e89cb73ac2bf04bd706858c915a6a12
Author: Jarek Potiuk 
AuthorDate: Sun May 22 14:13:44 2022 +0200

Upgrade `pip` to 22.1.1 version (just released) (#23854)
---
 Dockerfile  | 4 ++--
 Dockerfile.ci   | 4 ++--
 IMAGES.rst  | 2 +-
 dev/TRACKING_BACKTRACKING_ISSUES.md | 2 +-
 docs/docker-stack/build-arg-ref.rst | 2 +-
 scripts/ci/libraries/_initialization.sh | 2 +-
 scripts/docker/common.sh| 2 +-
 7 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 315a8908dc..4c567927ab 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -48,7 +48,7 @@ ARG AIRFLOW_VERSION="2.3.0"
 
 ARG PYTHON_BASE_IMAGE="python:3.7-slim-bullseye"
 
-ARG AIRFLOW_PIP_VERSION=22.1.0
+ARG AIRFLOW_PIP_VERSION=22.1.1
 ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow;
 ARG 
AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md;
 
@@ -367,7 +367,7 @@ function common::get_airflow_version_specification() {
 function common::override_pip_version_if_needed() {
 if [[ -n ${AIRFLOW_VERSION} ]]; then
 if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; 
then
-export AIRFLOW_PIP_VERSION="22.1.0"
+export AIRFLOW_PIP_VERSION="22.1.1"
 fi
 fi
 }
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 14b6a49d82..457eaf9aa4 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -327,7 +327,7 @@ function common::get_airflow_version_specification() {
 function common::override_pip_version_if_needed() {
 if [[ -n ${AIRFLOW_VERSION} ]]; then
 if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; 
then
-export AIRFLOW_PIP_VERSION="22.1.0"
+export AIRFLOW_PIP_VERSION="22.1.1"
 fi
 fi
 }
@@ -1195,7 +1195,7 @@ ARG AIRFLOW_CI_BUILD_EPOCH="3"
 ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
 # By default in the image, we are installing all providers when installing 
from sources
 ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG AIRFLOW_PIP_VERSION=22.1.0
+ARG AIRFLOW_PIP_VERSION=22.1.1
 # Setup PIP
 # By default PIP install run without cache to make image smaller
 ARG PIP_NO_CACHE_DIR="true"
diff --git a/IMAGES.rst b/IMAGES.rst
index 85427e2371..0a16bf18b7 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -475,7 +475,7 @@ The following build arguments (``--build-arg`` in docker 
build command) can be u
 | ``ADDITIONAL_RUNTIME_APT_ENV``   |   
   | Additional env variables defined |
 |  |   
   | when installing runtime deps |
 
+--+--+--+
-| ``AIRFLOW_PIP_VERSION``  | ``22.1.0``
   | PIP version used.|
+| ``AIRFLOW_PIP_VERSION``  | ``22.1.1``
   | PIP version used.|
 
+--+--+--+
 | ``PIP_PROGRESS_BAR`` | ``on``
   | Progress bar for PIP installation|
 
+--+--+--+
diff --git a/dev/TRACKING_BACKTRACKING_ISSUES.md 
b/dev/TRACKING_BACKTRACKING_ISSUES.md
index ca3747e6fb..57de81e042 100644
--- a/dev/TRACKING_BACKTRACKING_ISSUES.md
+++ b/dev/TRACKING_BACKTRACKING_ISSUES.md
@@ -42,7 +42,7 @@ image build jobs in CI.
 An example of such issue is described 
[here](https://github.com/pypa/pip/issues/10924).
 
 Unfortunately the problem is that in such cases, it is not possible to figure 
out what caused the
-problem from `pip` output (state as of `pip` 22.1.0).
+problem from `pip` output (state as of `pip` 22.1.1).
 
 There are a number of issues in `pip` that describe the issue, and some 
backtracking reasons have been already
 tracked down and fixed by `pip` maintainers, but this is a difficult problem 
to solve and it is likely it
diff --git a/docs/docker-stack/build-arg-ref.rst 
b/docs/docker-stack/build-arg-ref.rst
index d7edc2a386..74078c84dd 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you 
want to build a 

[GitHub] [airflow] potiuk merged pull request #23854: Upgrade `pip` to 22.1.1 version (just released)

[GitBox]

potiuk merged PR #23854:
URL: https://github.com/apache/airflow/pull/23854


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23854: Upgrade `pip` to 22.1.1 version (just released)

[GitBox]

github-actions[bot] commented on PR #23854:
URL: https://github.com/apache/airflow/pull/23854#issuecomment-1133882322

   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] don1uppa commented on issue #22863: No clear celery configuration example using sqs

[GitBox]

don1uppa commented on issue #22863:
URL: https://github.com/apache/airflow/issues/22863#issuecomment-1133879172

   I'll review.
   
   On Sun, May 22, 2022, 7:13 AM Jarek Potiuk ***@***.***> wrote:
   
   > @potiuk  I’ll do this :) please assign me
   >
   > Fantastic! Assigned you!
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23827: Add better feedback to Breeze users about expected action timing

[GitBox]

potiuk commented on PR #23827:
URL: https://github.com/apache/airflow/pull/23827#issuecomment-1133878266

   Would love to get it merged soon, to get Breeze users better feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #23854: Upgrade `pip` to 22.1.1 version (just released)

[GitBox]

potiuk commented on PR #23854:
URL: https://github.com/apache/airflow/pull/23854#issuecomment-1133878103

   Looks good with the new `pip`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix DataprocJobBaseOperator not being compatible with dotted names (#23439). (#23791)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a43e98d050 Fix DataprocJobBaseOperator not being compatible with 
dotted names (#23439). (#23791)
a43e98d050 is described below

commit a43e98d05047d9c4d5a7778bcb10efc4bdef7a01
Author: Guilherme Martins Crocetti 
<24530683+gmcroce...@users.noreply.github.com>
AuthorDate: Sun May 22 08:43:21 2022 -0300

Fix DataprocJobBaseOperator not being compatible with dotted names 
(#23439). (#23791)

* job_name parameter is now sanitized, replacing dots by underscores.
---
 airflow/providers/google/cloud/hooks/dataproc.py   |  7 +++--
 .../providers/google/cloud/hooks/test_dataproc.py  | 32 +---
 .../google/cloud/operators/test_dataproc.py| 35 +-
 3 files changed, 52 insertions(+), 22 deletions(-)

diff --git a/airflow/providers/google/cloud/hooks/dataproc.py 
b/airflow/providers/google/cloud/hooks/dataproc.py
index 784b4dd169..d870d80726 100644
--- a/airflow/providers/google/cloud/hooks/dataproc.py
+++ b/airflow/providers/google/cloud/hooks/dataproc.py
@@ -58,7 +58,7 @@ class DataProcJobBuilder:
 job_type: str,
 properties: Optional[Dict[str, str]] = None,
 ) -> None:
-name = task_id + "_" + str(uuid.uuid4())[:8]
+name = f"{task_id.replace('.', '_')}_{uuid.uuid4()!s:.8}"
 self.job_type = job_type
 self.job = {
 "job": {
@@ -175,11 +175,12 @@ class DataProcJobBuilder:
 
 def set_job_name(self, name: str) -> None:
 """
-Set Dataproc job name.
+Set Dataproc job name. Job name is sanitized, replacing dots by 
underscores.
 
 :param name: Job name.
 """
-self.job["job"]["reference"]["job_id"] = name + "_" + 
str(uuid.uuid4())[:8]
+sanitized_name = f"{name.replace('.', '_')}_{uuid.uuid4()!s:.8}"
+self.job["job"]["reference"]["job_id"] = sanitized_name
 
 def build(self) -> Dict:
 """
diff --git a/tests/providers/google/cloud/hooks/test_dataproc.py 
b/tests/providers/google/cloud/hooks/test_dataproc.py
index 2356f3cc00..a8d91daf40 100644
--- a/tests/providers/google/cloud/hooks/test_dataproc.py
+++ b/tests/providers/google/cloud/hooks/test_dataproc.py
@@ -23,6 +23,7 @@ from unittest.mock import ANY
 import pytest
 from google.api_core.gapic_v1.method import DEFAULT
 from google.cloud.dataproc_v1 import JobStatus
+from parameterized import parameterized
 
 from airflow.exceptions import AirflowException
 from airflow.providers.google.cloud.hooks.dataproc import DataprocHook, 
DataProcJobBuilder
@@ -472,27 +473,28 @@ class TestDataProcJobBuilder(unittest.TestCase):
 properties={"test": "test"},
 )
 
+@parameterized.expand([TASK_ID, f"group.{TASK_ID}"])
 @mock.patch(DATAPROC_STRING.format("uuid.uuid4"))
-def test_init(self, mock_uuid):
+def test_init(self, job_name, mock_uuid):
 mock_uuid.return_value = "uuid"
 properties = {"test": "test"}
-job = {
+expected_job_id = f"{job_name}_{mock_uuid.return_value}".replace(".", 
"_")
+expected_job = {
 "job": {
 "labels": {"airflow-version": AIRFLOW_VERSION},
 "placement": {"cluster_name": CLUSTER_NAME},
-"reference": {"job_id": TASK_ID + "_uuid", "project_id": 
GCP_PROJECT},
+"reference": {"job_id": expected_job_id, "project_id": 
GCP_PROJECT},
 "test": {"properties": properties},
 }
 }
 builder = DataProcJobBuilder(
 project_id=GCP_PROJECT,
-task_id=TASK_ID,
+task_id=job_name,
 cluster_name=CLUSTER_NAME,
 job_type="test",
 properties=properties,
 )
-
-assert job == builder.job
+assert expected_job == builder.job
 
 def test_add_labels(self):
 labels = {"key": "value"}
@@ -559,14 +561,22 @@ class TestDataProcJobBuilder(unittest.TestCase):
 self.builder.set_python_main(main)
 assert main == 
self.builder.job["job"][self.job_type]["main_python_file_uri"]
 
+@parameterized.expand(
+[
+("simple", "name"),
+("name with underscores", "name_with_dash"),
+("name with dot", "group.name"),
+("name with dot and underscores", "group.name_with_dash"),
+]
+)
 @mock.patch(DATAPROC_STRING.format("uuid.uuid4"))
-def test_set_job_name(self, mock_uuid):
+def test_set_job_name(self, name, job_name, mock_uuid):
 uuid = "test_uuid"
+expected_job_name = f"{job_name}_{uuid[:8]}".replace(".", "_")
 mock_uuid.return_value = uuid
-name = "name"
-self.builder.set_job_name(name)
-name += "_" + uuid[:8]
-assert name == 

[GitHub] [airflow] github-actions[bot] commented on pull request #23791: Fix DataprocJobBaseOperator not compatible with TaskGroups

[GitBox]

github-actions[bot] commented on PR #23791:
URL: https://github.com/apache/airflow/pull/23791#issuecomment-1133877102

   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest main or amend the last commit of the PR, and push 
it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #23439: DataprocJobBaseOperator not compatible with TaskGroups

[GitBox]

potiuk closed issue #23439: DataprocJobBaseOperator not compatible with 
TaskGroups
URL: https://github.com/apache/airflow/issues/23439


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #23791: Fix DataprocJobBaseOperator not compatible with TaskGroups

[GitBox]

potiuk merged PR #23791:
URL: https://github.com/apache/airflow/pull/23791


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org