[ https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481468#comment-17481468 ]
Yun Gao commented on FLINK-18356: --------------------------------- With some more observation, it seems (sorry I still have not got the final result yet) # Currently the test mostly failed in flink-table/flink-table-planner module. The tests of this module contains two parts, the tests and the integration tests. The failure always happens in the integration tests parts. # In Azure there are two parallel surefire test processes. Since the flink-table-planner module has set reuseForks = true, it means the same two processes would be used to run all the integration tests. Thus if we do not have correctly cleanup or some cases have memory leaking, the memory used would keep increasing. # By add some print statements to the watchdog process: [https://github.com/apache/flink/pull/18486,] from the result [https://dev.azure.com/gaoyunhaii/gaoyun-flink/_build/results?buildId=562&view=logs&j=43a593e7-535d-554b-08cc-244368da36b4&t=82d122c0-8bbf-56f3-4c0d-8e3d69630d0f] it seems the total memory is indeed 7G as Dawid pointed out, and the memory usage is keeping increasing. # Fortunately the case could be reproduced locally: by first run _mvn clean install_ then run {_}mvn -Dflink.forkCount=2 -Dcheckstyle.skip=true verify -pl flink-table/flink-table-planner{_}, the memory of the two processes are keeping increasing, the maximum memory required is 4G for each process. This is also wired since we have limit the heap to 2G. By adding -XX:NativeMemoryTracking=detail to the surefire plugin JVM options, the memory tracking result at the end of the tests are as follows. {code:java} Native Memory Tracking:Total: reserved=5199583KB +38339KB, committed=3802831KB +44371KB- Java Heap (reserved=2097152KB, committed=1575936KB) (mmap: reserved=2097152KB, committed=1575936KB) - Class (reserved=2342856KB +37546KB, committed=1534368KB +42666KB) (classes #193700 +5400) (malloc=38856KB +682KB #351017 +7653) (mmap: reserved=2304000KB +36864KB, committed=1495512KB +41984KB) - Thread (reserved=48453KB -969KB, committed=48453KB -969KB) (thread #48 -1) (stack: reserved=48188KB -1028KB, committed=48188KB -1028KB) (malloc=146KB -3KB #250 -5) (arena=119KB +63 #90 -2) - Code (reserved=287969KB +111KB, committed=244357KB +1023KB) (malloc=38369KB +111KB #69847 +655) (mmap: reserved=249600KB, committed=205988KB +912KB) - GC (reserved=146916KB +24KB, committed=127580KB +24KB) (malloc=36324KB +24KB #148561 +842) (mmap: reserved=110592KB, committed=91256KB) - Compiler (reserved=442KB, committed=442KB) (malloc=312KB #8705 +2) (arena=131KB #7) - Internal (reserved=177788KB +890KB, committed=177784KB +890KB) (malloc=177752KB +890KB #316112 +8088) (mmap: reserved=36KB, committed=32KB) - Symbol (reserved=43952KB +93KB, committed=43952KB +93KB) (malloc=42122KB +93KB #393043 +1171) (arena=1830KB #1) - Native Memory Tracking (reserved=21378KB +675KB, committed=21378KB +675KB) (malloc=1011KB +314KB #14713 +4635) (tracking overhead=20367KB +361KB) - Arena Chunk (reserved=28582KB -31KB, committed=28582KB -31KB) (malloc=28582KB -31KB) - Unknown (reserved=4096KB, committed=0KB) (mmap: reserved=4096KB, committed=0KB) {code} It seems the heap part and the classes part contributes to most of the memory consumption. > Exit code 137 returned from process > ----------------------------------- > > Key: FLINK-18356 > URL: https://issues.apache.org/jira/browse/FLINK-18356 > Project: Flink > Issue Type: Bug > Components: Build System / Azure Pipelines, Tests > Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0 > Reporter: Piotr Nowojski > Assignee: Dawid Wysakowicz > Priority: Blocker > Labels: pull-request-available, test-stability > Fix For: 1.15.0 > > > {noformat} > ============================= test session starts > ============================== > platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1 > cachedir: .tox/py37-cython/.pytest_cache > rootdir: /__w/3/s/flink-python > collected 568 items > pyflink/common/tests/test_configuration.py .......... [ > 1%] > pyflink/common/tests/test_execution_config.py ....................... [ > 5%] > pyflink/dataset/tests/test_execution_environment.py . > ##[error]Exit code 137 returned from process: file name '/bin/docker', > arguments 'exec -i -u 1002 > 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb > /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'. > Finishing: Test - python > {noformat} > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3 -- This message was sent by Atlassian Jira (v8.20.1#820001)