GitHub user potiuk added a comment to the discussion: Airflow 3.1.3 -
DagProccessor CPU throttling
Interesting discussion. But i am not sure what "problem" you are desribing. You
describe change in behaviour, for sure, but not necesarily problem,
Yes, if you limit CPU for DagProcess and it will want to use more of it, it
will get throttled. But maybe simply it means that DagFileprocessor is just
more efficient in parallelims in 3.1.3 and simply your throtling does not let
you utilise it's new capabilities now - and for example be more efficient in
parsing things in parallel.
There are various configuration option you can set to control parallelism (see
configuration of dag Processor) and if your goal is to limit how many CPUS it
uses, maybe you should use those options, I don't see how just "presence" of
throttling without explaining your reasoning of how you deliberately configured
it for throttling, what your decionse were when you tested were - so this post
does not classify as "problem" (without having an actual problem statement) -
rather than merely an "observation".
If you would like to discuss your assumptions, calculations and results of
tests showing that this throtting is something that does not come from simply
the fact that you set specific throttling levels - feel free to discuss it
further here, but you need to specify what your problem statement is other than
("things changed") - because this is the only reasonable thing that you can see
from your graphs.
Yes. They changed .. . So ?
GitHub link:
https://github.com/apache/airflow/discussions/58403#discussioncomment-14992429
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
