o-nikolas commented on code in PR #32572:
URL: https://github.com/apache/airflow/pull/32572#discussion_r1261820587
##########
airflow/config_templates/config.yml:
##########
@@ -2451,14 +2451,15 @@ scheduler:
max_tis_per_query:
description: |
This changes the batch size of queries in the scheduling main loop.
+ This should not be greater than ``core.parallelism``.
If this is too high, SQL query performance may be impacted by
complexity of query predicate, and/or excessive locking.
Additionally, you may hit the maximum allowable query length for your
db.
- Set this to 0 for no limit (not advised)
+ Setting this to 0 for using the value of ``core.parallelism``
Review Comment:
This wording is confusing. I think it's trying to say this?:
```suggestion
Setting this to 0 will use the value of ``core.parallelism``
```
##########
airflow/configuration.py:
##########
@@ -378,6 +379,27 @@ def validate(self):
self._upgrade_postgres_metastore_conn()
self.is_validated = True
+ def _validate_max_tis_per_query(self) -> None:
+ """
+ Check if config ``scheduler.max_tis_per_query`` is not greater than
``core.parallelism``.
+ If not met, a warning message printed, to guide user for best practice.
+
+ More info: https://github.com/apache/airflow/pull/29602
+
+ """
+ # TODO: this type of checking can be generalized like the deprecate
checking
+ max_tis_per_query = self.getint("scheduler", "max_tis_per_query")
+ parallelism = self.getint("core", "parallelism")
+
+ if max_tis_per_query > parallelism:
+ warnings.warn(
+ f"Configure
`scheduler.max_tis_per_query`(value:{max_tis_per_query}) "
+ f"should NOT be greater than
`core.parallelism`(value:{parallelism}). "
+ f"Now, SchedulerJob will take up to `core.parallelism` as the
query batch "
+ f"size when enqueue TaskInstances.",
Review Comment:
Do we need this detail? I find the wording a bit confusing, and perhaps it
doesn't really matter for the user to know the justification (which likely wont
make any sense to them). If anything, maybe just a link to the rst doc section
for a more detailed explanation.
##########
airflow/config_templates/default_airflow.cfg:
##########
@@ -1273,11 +1273,12 @@ catchup_by_default = True
ignore_first_depends_on_past_by_default = True
# This changes the batch size of queries in the scheduling main loop.
+# This should not be greater than ``core.parallelism``.
# If this is too high, SQL query performance may be impacted by
# complexity of query predicate, and/or excessive locking.
# Additionally, you may hit the maximum allowable query length for your db.
-# Set this to 0 for no limit (not advised)
-max_tis_per_query = 512
+# Setting this to 0 for using the value of ``core.parallelism``
Review Comment:
```suggestion
# Setting this to 0 will use the value of ``core.parallelism``
```
##########
docs/apache-airflow/administration-and-deployment/scheduler.rst:
##########
@@ -368,11 +368,12 @@ However, you can also look at other
non-performance-related scheduler configurat
The scheduler will list and sort the DAG files to decide the parsing order.
- :ref:`config:scheduler__max_tis_per_query`
- The batch size of queries in the scheduling main loop. If this is too high,
SQL query
- performance may be impacted by complexity of query predicate, and/or
excessive locking.
+ The batch size of queries in the scheduling main loop. This should not be
greater than
+ ``core.parallelism``.If this is too high, SQL query performance may be
impacted by
+ complexity of query predicate, and/or excessive locking.
Review Comment:
```suggestion
``core.parallelism``. If this is too high then SQL query performance may
be impacted by
complexity of query predicate and/or excessive locking.
```
##########
docs/apache-airflow/administration-and-deployment/scheduler.rst:
##########
@@ -368,11 +368,12 @@ However, you can also look at other
non-performance-related scheduler configurat
The scheduler will list and sort the DAG files to decide the parsing order.
- :ref:`config:scheduler__max_tis_per_query`
- The batch size of queries in the scheduling main loop. If this is too high,
SQL query
- performance may be impacted by complexity of query predicate, and/or
excessive locking.
+ The batch size of queries in the scheduling main loop. This should not be
greater than
+ ``core.parallelism``.If this is too high, SQL query performance may be
impacted by
+ complexity of query predicate, and/or excessive locking.
Additionally, you may hit the maximum allowable query length for your db.
- Set this to 0 for no limit (not advised).
+ Setting this to 0 for using the value of ``core.parallelism``.
Review Comment:
```suggestion
Setting this to 0 will use the value of ``core.parallelism``.
```
##########
airflow/configuration.py:
##########
@@ -378,6 +379,27 @@ def validate(self):
self._upgrade_postgres_metastore_conn()
self.is_validated = True
+ def _validate_max_tis_per_query(self) -> None:
+ """
+ Check if config ``scheduler.max_tis_per_query`` is not greater than
``core.parallelism``.
+ If not met, a warning message printed, to guide user for best practice.
Review Comment:
```suggestion
Check if config ``scheduler.max_tis_per_query`` is not greater than
``core.parallelism``.
If not met, a warning message is printed to guide the user to
correct it.
```
##########
airflow/configuration.py:
##########
@@ -378,6 +379,27 @@ def validate(self):
self._upgrade_postgres_metastore_conn()
self.is_validated = True
+ def _validate_max_tis_per_query(self) -> None:
+ """
+ Check if config ``scheduler.max_tis_per_query`` is not greater than
``core.parallelism``.
+ If not met, a warning message printed, to guide user for best practice.
+
+ More info: https://github.com/apache/airflow/pull/29602
+
+ """
+ # TODO: this type of checking can be generalized like the deprecate
checking
Review Comment:
Do we want to do this in this PR? Maybe cut an issue for the ToDo if not
(and link it here as well). ToDos in code rarely get done or noticed ever again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]