[
https://issues.apache.org/jira/browse/IMPALA-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sailesh Mukil resolved IMPALA-6792.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
> Appears to be a memory leak in orphaned fragments
> -------------------------------------------------
>
> Key: IMPALA-6792
> URL: https://issues.apache.org/jira/browse/IMPALA-6792
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.12.0
> Reporter: Mostafa Mokhtar
> Assignee: Sailesh Mukil
> Priority: Critical
> Fix For: Impala 2.12.0
>
> Attachments: Screen Shot 2018-04-02 at 12.08.54 PM.png, growth.pdf,
> memory_leak.txt
>
>
> On Impala daemons that have orphaned fragments it appears that memory
> continued to grow after query got cancelled, indicating of a memory leak.
> Memz page indicate that some fragments continue to run.
> {code}
> Memory Usage
> Memory consumption / limit: 101.48 GB / 64.00 GB
> Breakdown
> Process: memory limit exceeded. Limit=64.00 GB Total=101.51 GB Peak=101.51 GB
> Buffer Pool: Free Buffers: Total=0
> Buffer Pool: Clean Pages: Total=0
> Buffer Pool: Unused Reservation: Total=-71.09 MB
> Data Stream Service Queue: Limit=3.20 GB Total=0 Peak=875.26 KB
> Data Stream Manager Early RPCs: Total=0 Peak=23.03 MB
> TCMalloc Overhead: Total=760.53 MB
> Free Disk IO Buffers: Total=1.24 GB Peak=1.30 GB
> RequestPool=root.default: Total=909.59 MB Peak=11.91 GB
> Query(6c4aa5355486f3f6:5125570400000000): Limit=6.00 GB Reservation=69.50
> MB ReservationLimit=4.80 GB OtherMemory=67.90 MB Total=137.40 MB Peak=400.25
> MB
> Unclaimed reservations: Reservation=34.00 MB OtherMemory=0 Total=34.00
> MB Peak=90.14 MB
> Fragment 6c4aa5355486f3f6:5125570400000845: Reservation=35.50 MB
> OtherMemory=31.87 MB Total=67.37 MB Peak=67.37 MB
> Runtime Filter Bank: Reservation=512.00 KB ReservationLimit=512.00 KB
> OtherMemory=0 Total=512.00 KB Peak=512.00 KB
> AGGREGATION_NODE (id=12): Total=4.00 KB Peak=4.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KB
> AGGREGATION_NODE (id=11): Total=17.12 KB Peak=17.12 KB
> Exprs: Total=17.12 KB Peak=17.12 KB
> HASH_JOIN_NODE (id=10): Reservation=17.00 MB OtherMemory=66.25 KB
> Total=17.06 MB Peak=17.06 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB
> Peak=13.12 KB
> EXCHANGE_NODE (id=19): Reservation=10.59 MB OtherMemory=6.69 MB
> Total=17.29 MB Peak=17.29 MB
> KrpcDeferredRpcs: Total=6.69 MB Peak=6.69 MB
> HASH_JOIN_NODE (id=9): Reservation=18.00 MB OtherMemory=34.25 KB
> Total=18.03 MB Peak=18.10 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=9): Total=13.12 KB Peak=45.12 KB
> Hash Join Builder (join_node_id=9) Exprs: Total=13.12 KB
> Peak=13.12 KB
> EXCHANGE_NODE (id=17): Reservation=14.44 MB OtherMemory=0 Total=14.44
> MB Peak=14.45 MB
> KrpcDeferredRpcs: Total=0 Peak=2.42 MB
> EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0
> Peak=776.00 KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=20): Total=3.12 KB Peak=3.12 KB
> CodeGen: Total=18.24 KB Peak=2.89 MB
> Fragment 6c4aa5355486f3f6:5125570400000047: Reservation=0 OtherMemory=0
> Total=0 Peak=41.25 MB
> HDFS_SCAN_NODE (id=4): Total=0 Peak=18.75 MB
> KrpcDataStreamSender (dst_id=17): Total=0 Peak=22.19 MB
> CodeGen: Total=0 Peak=222.50 KB
> Fragment 6c4aa5355486f3f6:512557040000033f: Reservation=0 OtherMemory=0
> Total=0 Peak=241.80 MB
> HASH_JOIN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=2.00
> MB
> Hash Join Builder (join_node_id=8): Total=0 Peak=21.12 KB
> HASH_JOIN_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=1.99
> MB
> Hash Join Builder (join_node_id=7): Total=0 Peak=21.12 KB
> HASH_JOIN_NODE (id=6): Reservation=0 OtherMemory=0 Total=0
> Peak=128.08 MB
> Hash Join Builder (join_node_id=6): Total=0 Peak=21.12 KB
> HDFS_SCAN_NODE (id=0): Total=0 Peak=106.87 MB
> EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=2.46
> MB
> KrpcDeferredRpcs: Total=0 Peak=388.27 KB
> EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=18): Total=0 Peak=2.61 MB
> CodeGen: Total=0 Peak=3.61 MB
> Fragment 6c4aa5355486f3f6:51255704000002e8: Reservation=0 OtherMemory=0
> Total=0 Peak=6.71 MB
> HDFS_SCAN_NODE (id=2): Total=0 Peak=3.44 MB
> KrpcDataStreamSender (dst_id=14): Total=0 Peak=3.23 MB
> CodeGen: Total=0 Peak=241.50 KB
> Fragment 6c4aa5355486f3f6:51255704000005ea: Reservation=0
> OtherMemory=36.02 MB Total=36.02 MB Peak=46.83 MB
> HDFS_SCAN_NODE (id=5): Total=9.55 MB Peak=18.17 MB
> KrpcDataStreamSender (dst_id=19): Total=26.46 MB Peak=28.87 MB
> CodeGen: Total=234.00 B Peak=52.50 KB
> Query(b44c5962ad16140:4bd217e100000000): Limit=6.00 GB Reservation=660.28
> MB ReservationLimit=4.80 GB OtherMemory=111.92 MB Total=772.20 MB Peak=778.99
> MB
> Unclaimed reservations: Reservation=3.94 MB OtherMemory=0 Total=3.94 MB
> Peak=120.66 MB
> Fragment b44c5962ad16140:4bd217e100000bbb: Reservation=0
> OtherMemory=49.74 KB Total=49.74 KB Peak=1.41 MB
> SORT_NODE (id=16): Total=8.00 KB Peak=8.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KB
> AGGREGATION_NODE (id=27): Total=25.12 KB Peak=25.12 KB
> Exprs: Total=25.12 KB Peak=25.12 KB
> EXCHANGE_NODE (id=26): Reservation=0 OtherMemory=0 Total=0 Peak=0
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=28): Total=3.88 KB Peak=3.88 KB
> CodeGen: Total=4.74 KB Peak=1.37 MB
> Fragment b44c5962ad16140:4bd217e100000934: Reservation=332.39 MB
> OtherMemory=15.81 MB Total=348.20 MB Peak=358.59 MB
> Runtime Filter Bank: Reservation=2.52 MB ReservationLimit=2.52 MB
> OtherMemory=0 Total=2.52 MB Peak=2.52 MB
> AGGREGATION_NODE (id=15): Total=25.12 KB Peak=25.12 KB
> Exprs: Total=25.12 KB Peak=25.12 KB
> HASH_JOIN_NODE (id=14): Reservation=1.94 MB OtherMemory=82.25 KB
> Total=2.02 MB Peak=2.03 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=14): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=14) Exprs: Total=13.12 KB
> Peak=13.12 KB
> HASH_JOIN_NODE (id=13): Reservation=1.94 MB OtherMemory=74.25 KB
> Total=2.01 MB Peak=2.03 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=13): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=13) Exprs: Total=13.12 KB
> Peak=13.12 KB
> HASH_JOIN_NODE (id=12): Reservation=160.00 MB OtherMemory=134.25 KB
> Total=160.13 MB Peak=160.15 MB
> Exprs: Total=47.12 KB Peak=47.12 KB
> Hash Join Builder (join_node_id=12): Total=47.12 KB Peak=55.12 KB
> Hash Join Builder (join_node_id=12) Exprs: Total=47.12 KB
> Peak=47.12 KB
> HASH_JOIN_NODE (id=11): Reservation=160.00 MB OtherMemory=62.25 KB
> Total=160.06 MB Peak=160.08 MB
> Exprs: Total=17.12 KB Peak=17.12 KB
> Hash Join Builder (join_node_id=11): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=11) Exprs: Total=13.12 KB
> Peak=13.12 KB
> HASH_JOIN_NODE (id=10): Reservation=6.00 MB OtherMemory=54.25 KB
> Total=6.05 MB Peak=6.07 MB
> Exprs: Total=17.12 KB Peak=17.12 KB
> Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB
> Peak=13.12 KB
> EXCHANGE_NODE (id=20): Reservation=14.34 MB OtherMemory=0 Total=14.34
> MB Peak=14.34 MB
> KrpcDeferredRpcs: Total=0 Peak=1.16 MB
> EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=2.11
> MB
> KrpcDeferredRpcs: Total=0 Peak=232.95 KB
> EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=10.97
> MB
> KrpcDeferredRpcs: Total=0 Peak=805.37 KB
> EXCHANGE_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=13.84
> MB
> KrpcDeferredRpcs: Total=0 Peak=1.69 MB
> EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=40.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=8.10 KB
> EXCHANGE_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=26): Total=1021.05 KB Peak=1021.05 KB
> CodeGen: Total=48.00 KB Peak=8.65 MB
> Fragment b44c5962ad16140:4bd217e10000054e: Reservation=321.00 MB
> OtherMemory=20.82 MB Total=341.82 MB Peak=352.83 MB
> Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB
> OtherMemory=0 Total=1.00 MB Peak=1.00 MB
> HASH_JOIN_NODE (id=9): Reservation=320.00 MB OtherMemory=60.25 KB
> Total=320.06 MB Peak=320.07 MB
> Exprs: Total=22.12 KB Peak=22.12 KB
> Hash Join Builder (join_node_id=9): Total=22.12 KB Peak=30.12 KB
> Hash Join Builder (join_node_id=9) Exprs: Total=22.12 KB
> Peak=22.12 KB
> EXCHANGE_NODE (id=18): Reservation=12.48 MB OtherMemory=0 Total=12.48
> MB Peak=16.40 MB
> KrpcDeferredRpcs: Total=0 Peak=4.41 MB
> EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=4.35
> MB
> KrpcDeferredRpcs: Total=0 Peak=1.69 MB
> KrpcDataStreamSender (dst_id=20): Total=8.26 MB Peak=19.26 MB
> CodeGen: Total=8.02 KB Peak=1.34 MB
> Fragment b44c5962ad16140:4bd217e100000048: Reservation=2.95 MB
> OtherMemory=75.24 MB Total=78.19 MB Peak=86.44 MB
> Runtime Filter Bank: Reservation=1.02 MB ReservationLimit=1.02 MB
> OtherMemory=0 Total=1.02 MB Peak=1.02 MB
> HASH_JOIN_NODE (id=8): Reservation=1.94 MB OtherMemory=79.25 KB
> Total=2.01 MB Peak=2.01 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB
> Peak=13.12 KB
> HDFS_SCAN_NODE (id=0): Total=63.00 MB Peak=63.01 MB
> EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=18): Total=12.09 MB Peak=20.43 MB
> CodeGen: Total=8.66 KB Peak=1.50 MB
> Fragment b44c5962ad16140:4bd217e1000002f3: Reservation=0 OtherMemory=0
> Total=0 Peak=81.41 MB
> HDFS_SCAN_NODE (id=1): Total=0 Peak=60.83 MB
> KrpcDataStreamSender (dst_id=19): Total=0 Peak=19.08 MB
> CodeGen: Total=0 Peak=335.00 KB
> Fragment b44c5962ad16140:4bd217e1000007ef: Reservation=0 OtherMemory=0
> Total=0 Peak=26.00 MB
> HDFS_SCAN_NODE (id=5): Total=0 Peak=6.29 MB
> KrpcDataStreamSender (dst_id=21): Total=0 Peak=19.66 MB
> CodeGen: Total=0 Peak=241.50 KB
> Untracked Memory: Total=98.71 GB
> tcmalloc
> ------------------------------------------------
> MALLOC: 108196856848 (103184.6 MiB) Bytes in use by application
> MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> MALLOC: + 625752432 ( 596.8 MiB) Bytes in central cache freelist
> MALLOC: + 25029120 ( 23.9 MiB) Bytes in transfer cache freelist
> MALLOC: + 146667648 ( 139.9 MiB) Bytes in thread cache freelists
> MALLOC: + 449097920 ( 428.3 MiB) Bytes in malloc metadata
> MALLOC: ------------
> MALLOC: = 109443403968 (104373.4 MiB) Actual memory used (physical + swap)
> MALLOC: + 221184 ( 0.2 MiB) Bytes released to OS (aka unmapped)
> MALLOC: ------------
> MALLOC: = 109443625152 (104373.6 MiB) Virtual address space used
> MALLOC:
> MALLOC: 6575796 Spans in use
> MALLOC: 380 Thread heaps in use
> MALLOC: 8192 Tcmalloc page size
> ------------------------------------------------
> Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
> Bytes released to the OS take up virtual address space but no physical memory.
> System
> Physical Memory: 125.52 GB
> Transparent Huge Pages Config:
> enabled: always [madvise] never
> defrag: always [madvise] never
> khugepaged defrag: 1
> Process and system memory metrics
> Name Value Description
> memory.anon-huge-page-bytes 640.00 MB Total bytes of anonymous
> (a.k.a. transparent) huge pages used by this process.
> memory.mapped-bytes 137.22 GB Total bytes of memory mappings in this
> process (the virtual memory size).
> memory.num-maps 4236 Total number of memory mappings in this process.
> memory.rss 68.40 GB Resident set size (RSS) of this process,
> including TCMalloc, buffer pool and Jvm.
> memory.thp.defrag always [madvise] never The system-wide 'defrag'
> setting for Transparent Huge Pages.
> memory.thp.enabled always [madvise] never The system-wide 'enabled'
> setting for Transparent Huge Pages.
> memory.thp.khugepaged-defrag 1 The system-wide 'defrag' setting for
> khugepaged.
> memory.total-used 101.51 GB Total memory currently used by TCMalloc
> and buffer pool.
> Buffer pool memory metrics
> Name Value Description
> buffer-pool.clean-page-bytes 0 Total bytes of clean page memory cached
> in the buffer pool.
> buffer-pool.clean-pages 0 Total number of clean pages cached in
> the buffer pool.
> buffer-pool.clean-pages-limit 5.44 GB Limit on number of clean pages cached
> in the buffer pool.
> buffer-pool.free-buffer-bytes 0 Total bytes of free buffer memory
> cached in the buffer pool.
> buffer-pool.free-buffers 0 Total number of free buffers cached in
> the buffer pool.
> buffer-pool.limit 54.40 GB Maximum allowed bytes allocated by the
> buffer pool.
> buffer-pool.reserved 781.62 MB Total bytes of buffers reserved by
> Impala subsystems
> buffer-pool.system-allocated 710.53 MB Total buffer memory currently
> allocated by the buffer pool.
> buffer-pool.unused-reservation-bytes 71.09 MB Total bytes of buffer
> reservations by Impala subsystems that are currently unused
> JVM aggregate memory metrics
> Name Value Description
> jvm.total.committed-usage-bytes 1.73 GB Jvm total Committed Usage Bytes
> jvm.total.current-usage-bytes 385.19 MB Jvm total Current Usage Bytes
> jvm.total.init-usage-bytes 1.88 GB Jvm total Init Usage Bytes
> jvm.total.max-usage-bytes 31.23 GB Jvm total Max Usage Bytes
> jvm.total.peak-committed-usage-bytes 2.39 GB Jvm total Peak Committed Usage
> Bytes
> jvm.total.peak-current-usage-bytes 1.08 GB Jvm total Peak Current Usage
> Bytes
> jvm.total.peak-init-usage-bytes 1.88 GB Jvm total Peak Init Usage Bytes
> jvm.total.peak-max-usage-bytes 31.32 GB Jvm total Peak Max
> Usage Bytes
> JVM heap memory metrics
> Name Value Description
> jvm.heap.committed-usage-bytes 1.67 GB Jvm heap Committed Usage Bytes
> jvm.heap.current-usage-bytes 322.88 MB Jvm heap Current Usage Bytes
> jvm.heap.init-usage-bytes 1.96 GB Jvm heap Init Usage Bytes
> jvm.heap.max-usage-bytes 26.67 GB Jvm heap Max Usage Bytes
> jvm.heap.peak-committed-usage-bytes 0 Jvm heap Peak Committed Usage
> Bytes
> jvm.heap.peak-current-usage-bytes 0 Jvm heap Peak Current Usage
> Bytes
> jvm.heap.peak-init-usage-bytes 0 Jvm heap Peak Init Usage Bytes
> jvm.heap.peak-max-usage-bytes 0 Jvm heap Peak Max Usage Bytes
> JVM non-heap memory metrics
> Name Value Description
> jvm.non-heap.committed-usage-bytes 63.40 MB Jvm non-heap Committed
> Usage Bytes
> jvm.non-heap.current-usage-bytes 62.31 MB Jvm non-heap Current
> Usage Bytes
> jvm.non-heap.init-usage-bytes 2.44 MB Jvm non-heap Init Usage Bytes
> jvm.non-heap.max-usage-bytes -1.00 B Jvm non-heap Max Usage Bytes
> jvm.non-heap.peak-committed-usage-bytes 0 Jvm non-heap Peak
> Committed Usage Bytes
> jvm.non-heap.peak-current-usage-bytes 0 Jvm non-heap Peak Current Usage
> Bytes
> jvm.non-heap.peak-init-usage-bytes 0 Jvm non-heap Peak Init Usage
> Bytes
> jvm.non-heap.peak-max-usage-bytes 0 Jvm non-heap Peak Max Usage
> Bytes
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
