[
https://issues.apache.org/jira/browse/ARROW-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263828#comment-17263828
]
William Volkman commented on ARROW-9530:
----------------------------------------
I have some airflow tasks with python code that call pyarrow to create parquet
files. These open the parquet_writer, add some tables to them, close and the
handle goes out of scope so should be available for GC.
I've just shut them down and restarted. Though they have not done any work the
jemalloc_bg_thd processes are all taking 3 to 4 percent of the CPU.
The reason I commented was to point out that it is not as if they are not
taking any resources when idle. Look forward to the release with your runtime
controls.
> [C++] Add option to disable jemalloc background thread on Linux
> ---------------------------------------------------------------
>
> Key: ARROW-9530
> URL: https://issues.apache.org/jira/browse/ARROW-9530
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Rob Ambalu
> Priority: Minor
>
> We noticed that after we upgraded pyarrow to version 0.17.1 from 0.9.0 we now
> get two unwanted side effects just by linking in arrow libs into our c++
> build. We link arrow libs into one of our low level core libraries so now
> all of our applications are affected by these side effects:
> 1) a "jemalloc_bg_thd" thread is spawned on dlinit before we ever hit main
> 2) all our apps are now hitting valgrind leak warnings due to a (potential)
> leak in jemalloc code:
> ==33515== 656 bytes in 1 blocks are possibly lost in loss record 1 of 1
> ==33515== at 0x402E9EA: calloc (vg_replace_malloc.c:752)
> ==33515== by 0x4011F44: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
> ==33515== by 0x5DFF9C0: pthread_create@@GLIBC_2.2.5 (in
> /usr/lib64/libpthread-2.17.so)
> ==33515== by 0x589186B: je_arrow_private_je_pthread_create_wrapper
> (background_thread.c:48)
> ==33515== by 0x589186B: background_thread_create_signals_masked
> (background_thread.c:365)
> ==33515== by 0x589186B: background_thread_create_locked
> (background_thread.c:573)
> ==33515== by 0x5891A47: je_arrow_private_je_background_thread_create
> (background_thread.c:598)
> ==33515== by 0x400F502: _dl_init (in /usr/lib64/ld-2.17.so)
> ==33515== by 0x40011A9: ??? (in /usr/lib64/ld-2.17.so)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
