[
https://issues.apache.org/jira/browse/AIRFLOW-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887457#comment-16887457
]
Grant Nicholas edited comment on AIRFLOW-4593 at 7/17/19 10:29 PM:
-------------------------------------------------------------------
[~ash] I did some memory profiling of the 1.10.3 airflow scheduler.
```
// pyrasite-shell \{process_id_of_airflow_scheduler}
import tracemalloc
tracemalloc.start(25)
snapshot_1 = tracemalloc.take_snapshot()
// Wait a few hours
snapshot_2 = tracemalloc.take_snapshot()
top_stats = snapshot_2.compare_to(snapshot_1, 'lineno')
// Diff the memory allocation between the two snapshots
for stat in top_stats[0:10]:
print(stat)
// Print the traceback of the biggest memory block
for line in snapshot_2.statistics('traceback')[0].traceback.format():
print(line)
```
```
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:319:
size=226 KiB (+160 KiB), count=3543 (+2483), average=65 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:265:
size=37.1 KiB (+26.6 KiB), count=576 (+412), average=66 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py:90:
size=41.4 KiB (+11.4 KiB), count=539 (+143), average=79 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:812:
size=22.1 KiB (+8920 B), count=283 (+117), average=80 B
/usr/local/lib/python3.6/site-packages/airflow/utils/dag_processing.py:1002:
size=6896 B (+6896 B), count=78 (+78), average=88 B
/usr/local/lib/python3.6/site-packages/airflow/utils/dag_processing.py:988:
size=6600 B (+6600 B), count=75 (+75), average=88 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:733:
size=9352 B (+6440 B), count=167 (+115), average=56 B
/usr/local/lib/python3.6/site-packages/tabulate.py:1320: size=6952 B (+6248
B), count=79 (+71), average=88 B
/usr/local/lib/python3.6/weakref.py:335: size=4936 B (+3960 B), count=67
(+55), average=74 B
/usr/local/lib/python3.6/multiprocessing/process.py:81: size=18.9 KiB (-3624
B), count=268 (-49), average=72 B
/usr/local/lib/python3.6/tracemalloc.py:113: size=5456 B (+2816 B), count=62
(+32), average=88 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:2170:
size=14.5 KiB (-2480 B), count=12 (-2), average=1240 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:1513:
size=13.0 KiB (-2224 B), count=12 (-2), average=1112 B
/usr/local/lib/python3.6/os.py:409: size=26.7 KiB (+1952 B), count=28 (+2),
average=976 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:3041:
size=6216 B (-1848 B), count=37 (-11), average=168 B
/usr/local/lib/python3.6/tracemalloc.py:127: size=4320 B (+1840 B), count=54
(+23), average=80 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:1141:
size=10.4 KiB (-1776 B), count=36 (-6), average=296 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:417:
size=1944 B (+1752 B), count=21 (+19), average=93 B
/usr/local/lib/python3.6/weakref.py:356: size=4448 B (+1568 B), count=72
(+28), average=62 B
/usr/local/lib/python3.6/tracemalloc.py:462: size=24.7 KiB (+1536 B),
count=527 (+32), average=48 B
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py",
line 319
for obj_elem in elem[4]
```
It is a slow memory leak, but it looks like a bunch of sqlalchemy
ResultMetaData objects build up over time.
was (Author: grantnicholas):
[~ash] I did some memory profiling of the 1.10.3 airflow scheduler.
```
// pyrasite-shell \{process_id_of_airflow_scheduler}
import tracemalloc
tracemalloc.start(25)
snapshot_1 = tracemalloc.take_snapshot()
// Wait an hour
snapshot_2 = tracemalloc.take_snapshot()
top_stats = snapshot_2.compare_to(snapshot_1, 'lineno')
// Diff the memory allocation between the two snapshots
for stat in top_stats[0:10]:
print(stat)
// Print the traceback of the biggest memory block
for line in snapshot_2.statistics('traceback')[0].traceback.format():
print(line)
```
```
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:319:
size=226 KiB (+160 KiB), count=3543 (+2483), average=65 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:265:
size=37.1 KiB (+26.6 KiB), count=576 (+412), average=66 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py:90:
size=41.4 KiB (+11.4 KiB), count=539 (+143), average=79 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:812:
size=22.1 KiB (+8920 B), count=283 (+117), average=80 B
/usr/local/lib/python3.6/site-packages/airflow/utils/dag_processing.py:1002:
size=6896 B (+6896 B), count=78 (+78), average=88 B
/usr/local/lib/python3.6/site-packages/airflow/utils/dag_processing.py:988:
size=6600 B (+6600 B), count=75 (+75), average=88 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:733:
size=9352 B (+6440 B), count=167 (+115), average=56 B
/usr/local/lib/python3.6/site-packages/tabulate.py:1320: size=6952 B (+6248
B), count=79 (+71), average=88 B
/usr/local/lib/python3.6/weakref.py:335: size=4936 B (+3960 B), count=67
(+55), average=74 B
/usr/local/lib/python3.6/multiprocessing/process.py:81: size=18.9 KiB (-3624
B), count=268 (-49), average=72 B
/usr/local/lib/python3.6/tracemalloc.py:113: size=5456 B (+2816 B), count=62
(+32), average=88 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:2170:
size=14.5 KiB (-2480 B), count=12 (-2), average=1240 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py:1513:
size=13.0 KiB (-2224 B), count=12 (-2), average=1112 B
/usr/local/lib/python3.6/os.py:409: size=26.7 KiB (+1952 B), count=28 (+2),
average=976 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:3041:
size=6216 B (-1848 B), count=37 (-11), average=168 B
/usr/local/lib/python3.6/tracemalloc.py:127: size=4320 B (+1840 B), count=54
(+23), average=80 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py:1141:
size=10.4 KiB (-1776 B), count=36 (-6), average=296 B
/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:417:
size=1944 B (+1752 B), count=21 (+19), average=93 B
/usr/local/lib/python3.6/weakref.py:356: size=4448 B (+1568 B), count=72
(+28), average=62 B
/usr/local/lib/python3.6/tracemalloc.py:462: size=24.7 KiB (+1536 B),
count=527 (+32), average=48 B
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py",
line 319
for obj_elem in elem[4]
```
It is a slow memory leak, but it looks like a bunch of sqlalchemy
ResultMetaData objects build up over time.
> Memory leak in Airflow scheduler
> --------------------------------
>
> Key: AIRFLOW-4593
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4593
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.2
> Reporter: Nikhil SInghal
> Priority: Major
> Attachments: Screenshot 2019-05-30 at 3.19.33 PM.png
>
>
> We are running Apache Airflow on Kubernetes. When I see my Grafana Dashboard
> I see that the memory used is consistently increasing. Can anyone give me
> some pointers for how can I debug this or with existing issues/solutions
> related to this.
> This is a sharp increase in Airflow scheduler and slow increase for worker
> and webserver
> !Screenshot 2019-05-30 at 3.19.33 PM.png|width=100%!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
