a-meledin opened a new issue, #35267:
URL: https://github.com/apache/airflow/issues/35267
### Apache Airflow version
2.7.2
### What happened
We have a Dag with 116 tasks each having some number of MappedTasks. Overall
number of tasks instances is ~6000. Problem is that
VladimirYushkevich commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2110948819
I'm having similar issue with Dynamically Mapped Tasks for 2.8.4. What is a
best way to fix it?
--
This is an automated message from the Apache Git Service.
To respond
kaxil commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2112251512
@uranusjr Any ideas or other instance of such things you have noticed from
other users?
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
uranusjr commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2112291561
I don’t recall anything similar reported elsewhere. I can imagine the FOR
UPDATE part causing this though since it can saturate the database with all
workers doing it at the same
VladimirYushkevich commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2112691876
I managed to spot this problem a bit. When our DAG starts with 1000+
Dynamically Mapped Tasks then we are experiencing significant performance
issues. UI is slow, warnin
dstandish commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2113027901
This could be related to this PR, and looks like it was released in 2.9.1:
https://github.com/apache/airflow/pull/38914
We observed that when you have a bunch of mapp
VladimirYushkevich commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2115152013
@dstandish, thanks a lot. I see that after upgrading to `2.9.1` my issue is
gone. All queries are fast again and I don't see locks in DB.
--
This is an automated messa
dstandish commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2115608323
> @dstandish, thanks a lot. I see that after upgrading to `2.9.1` my issue
is gone. All queries are fast again and I don't see locks in DB.
Great , glad to hear it
--
T
kaxil closed issue #35267: MappedTasks: Very slow execution :: taskinstance.py
seems to be a bottleneck.
URL: https://github.com/apache/airflow/issues/35267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
kaxil commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2115631519
Going to close it as resolved in 2.9.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
potiuk commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2115694203
> > @dstandish, thanks a lot. I see that after upgrading to `2.9.1` my issue
is gone. All queries are fast again and I don't see locks in DB.
>
> Great , glad to hear it
VladimirYushkevich commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2117183182
I think `2.9.1` has another issue. We started to see:
```
2024-05-17 09:49:19.191 UTC [3586765]: [131-1]
db=airflow,user=airflow-cloudsqlpr...@service-dev.iam ERROR
kaxil commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-2117222836
Yes, please open a new issue and if possible please also add details of any
impact on performance or stability
--
This is an automated message from the Apache Git Service.
To respo
boring-cyborg[bot] commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1784812089
Thanks for opening your first issue here! Be sure to follow the issue
template! If you are willing to raise PR to address this issue please do so, no
need to wait for ap
Taragolis commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1784887119
> Invetigation required on taskinstance.py code.
Airflow is OSS and developed by different contributors (most of them also
users of Airflow), so this required if someone sp
phi-friday commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1785317720
I think you can limit it with
[config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-pool-size)
--
This is an automated message fro
potiuk commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1786255118
I think indeed we need more details an investigation and answers (from your
side @a-meledin ) .
* Is it only with mapped tasks in that number you see the problem?
* Have y
potiuk commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1786255698
> I think you can limit it with
[config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-pool-size)
That will not work - each task es
a-meledin commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1787027170
> I think you can limit it with
[config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-pool-size)
I. First, I installed pgbounce
a-meledin commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1789269209
Jarec,
Thank you for your answer. You could run the code above on 2.7.1 too.
Actually I don't have such a possibility now.
On Tue, 31 Oct 2023 at 23:15, Jarek Poti
potiuk commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1789323872
> Jarec, Thank you for your answer. You could run the code above on 2.7.1
too. Actually I don't have such a possibility now.
It's. not the code, it's the scale. I am contribut
21 matches
Mail list logo
