yikf opened a new pull request, #12740: URL: https://github.com/apache/gluten/pull/12740
## What changes are proposed in this pull request? `~ObjectStore()` checked `aliveObjects_.empty()` **outside** the lock and then read `aliveObjects_.rbegin()` **inside** the lock. While a store is being destructed, another thread can still call the static `ObjectStore::release()` (e.g. a script-transformation feed thread closing its output iterator during task teardown), which erases entries from `aliveObjects_` under the lock. A concurrent erase could empty the map between the unlocked check and `rbegin()`, so `rbegin()` dereferences an empty tree and crashes the JVM with a native SIGSEGV in `std::_Rb_tree_decrement`. Fix: move the empty-check under the lock so it is atomic with popping the last object. Surfaced as a flaky native crash in CI running `GlutenSparkScriptTransformationSuite`: - Failing job: https://github.com/apache/gluten/actions/runs/31084950810/job/92564768040?pr=12714 From `hs_err_pid*.log`: ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f3ae8f62f7e, pid=7077, tid=7655 # # JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS) # Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) # Problematic frame: # C [libgluten.so+0xcdbf7e] std::_Rb_tree_decrement(std::_Rb_tree_node_base*)+0xe # # Core dump will be written. Default location: Core dumps may be processed with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /__w/gluten/gluten/gluten-ut/spark40/core.7077) # # If you would like to submit a bug report, please visit: # https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- S U M M A R Y ------------ Command Line: -Dlog4j.configurationFile=file:src/test/resources/log4j2.properties -Dbasedir=/__w/gluten/gluten/gluten-ut/spark40 -Dspark.test.home=/opt/shims/spark40/spark_home/ -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dfile.encoding= UTF-8 org.scalatest.tools.Runner -R /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/classes /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/test-classes -l org.apache.spark.tags.ExtendedSQLTest org.apache.spark.tags.SlowHiveTest org.apache.gluten.tags.UDFTest org.apache.gluten.tags.SkipTest -w org.apache.spark.sql.execution -w org.apache.spark.sql.catalyst -w org.apache.spark.sql.errors -w org.apache.spark.sql.extension -o -u /__w/gluten/gluten/gluten-ut/spark40/target/surefire-reports/. Host: AMD EPYC 7763 64-Core Processor, 4 cores, 15G, CentOS Stream release 8 Time: Thu Aug 6 08:53:09 2026 UTC elapsed time: 23.585455 seconds (0d 0h 0m 23s) --------------- T H R E A D --------------- Current thread (0x00007f3b5000a060): JavaThread "Executor task launch worker for task 1.0 in stage 12.0 (TID 25)" daemon [_thread_in_native, id=7655, stack(0x00007f3abb3b6000,0x00007f3abb4b7000)] Stack: [0x00007f3abb3b6000,0x00007f3abb4b7000], sp=0x00007f3abb4b4548, free space=1017k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libgluten.so+0xcdbf7e] std::_Rb_tree_decrement(std::_Rb_tree_node_base*)+0xe Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 19294 org.apache.gluten.runtime.RuntimeJniWrapper.releaseRuntime(J)V (0 bytes) @ 0x00007f3b6d3363e9 [0x00007f3b6d3363a0+0x0000000000000049] J 19293 c1 org.apache.gluten.runtime.Runtime$RuntimeImpl.release()V (63 bytes) @ 0x00007f3b66c0bffc [0x00007f3b66c0ba60+0x000000000000059c] J 14720 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6960+0x00000008029b9890.apply$mcV$sp()V (8 bytes) @ 0x00007f3b663dc504 [0x00007f3b663dc3c0+0x0000000000000144] J 15357 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f3b6d0a315c [0x00007f3b6d0a3120+0x000000000000003c] J 14722 c1 org.apache.spark.task.TaskResourceRegistry.exclusiveLock(Lscala/Function0;)Ljava/lang/Object; (43 bytes) @ 0x00007f3b663dbe0c [0x00007f3b663dbc60+0x00000000000001ac] J 13925 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6959+0x00000008029b94c0.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007f3b6627ef0c [0x00007f3b6627e940+0x00000000000005cc] J 12394 c2 scala.collection.immutable.List.foreach(Lscala/Function1;)V (32 bytes) @ 0x00007f3b6cf578a0 [0x00007f3b6cf57820+0x0000000000000080] J 18048 c1 org.apache.spark.task.TaskResourceRegistry.$anonfun$releaseAll$3(Lorg/apache/spark/task/TaskResourceRegistry;Lscala/Tuple2;)V (56 bytes) @ 0x00007f3b662c05a4 [0x00007f3b662bf6e0+0x0000000000000ec4] J 15272 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6958+0x00000008029b90f0.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007f3b6657e1dc [0x00007f3b6657e040+0x000000000000019c] J 12394 c2 scala.collection.immutable.List.foreach(Lscala/Function1;)V (32 bytes) @ 0x00007f3b6cf578a0 [0x00007f3b6cf57820+0x0000000000000080] j org.apache.spark.task.TaskResourceRegistry.$anonfun$releaseAll$1(Lorg/apache/spark/task/TaskResourceRegistry;)V+31 j org.apache.spark.task.TaskResourceRegistry$$Lambda$6956+0x00000008029b8a48.apply$mcV$sp()V+4 J 15357 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f3b6d0a315c [0x00007f3b6d0a3120+0x000000000000003c] J 12463 c1 org.apache.spark.task.TaskResourceRegistry.lock(Lscala/Function0;)Ljava/lang/Object; (22 bytes) @ 0x00007f3b661f6fec [0x00007f3b661f6e60+0x000000000000018c] j org.apache.spark.task.TaskResourceRegistry.releaseAll()V+7 j org.apache.spark.task.TaskResources$$anon$3.onTaskCompletion(Lorg/apache/spark/TaskContext;)V+38 j org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(Lorg/apache/spark/TaskContextImpl;Lorg/apache/spark/util/TaskCompletionListener;)V+2 j org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(Lorg/apache/spark/TaskContextImpl;Lorg/apache/spark/util/TaskCompletionListener;)Ljava/lang/Object;+2 j org.apache.spark.TaskContextImpl$$Lambda$6955+0x00000008029b8678.apply(Ljava/lang/Object;)Ljava/lang/Object;+8 j org.apache.spark.TaskContextImpl.invokeListeners(Ljava/util/Stack;Ljava/lang/String;Lscala/Option;Lscala/Function1;)V+87 j org.apache.spark.TaskContextImpl.invokeTaskCompletionListeners(Lscala/Option;)V+15 j org.apache.spark.TaskContextImpl.markTaskCompleted(Lscala/Option;)V+32 j org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object;+39 j org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object;+258 j org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/collection/immutable/Map;Lscala/runtime/BooleanRef;)Ljava/lang/Object;+37 j org.apache.spark.executor.Executor$TaskRunner$$Lambda$7513+0x0000000802ad9248.apply()Ljava/lang/Object;+12 J 18605 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (225 bytes) @ 0x00007f3b66a10414 [0x00007f3b66a103a0+0x0000000000000074] J 16272 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(Lorg/apache/spark/util/SparkErrorUtils;Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (7 bytes) @ 0x00007f3b6676d6ec [0x00007f3b6676d640+0x00000000000000ac] j org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+3 j org.apache.spark.executor.Executor$TaskRunner.run()V+590 J 17287 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f3b651089c4 [0x00007f3b651080a0+0x0000000000000924] j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected] J 14300 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f3b66319f84 [0x00007f3b66319e40+0x0000000000000144] v ~StubRoutines::call_stub ``` ## How was this patch tested? flaky test, existed test to verify. ## Was this patch authored or co-authored using generative AI tooling? Yes, AI-assisted, Generated-by: Claude claude-opus-4-8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
