wangbing505 commented on issue #12456:
URL: https://github.com/apache/seatunnel/issues/12456#issuecomment-5889482372
@DanielLeens I have already provided the Create and Release classloader
logs for a specific job above.
I submitted a test job to the cluster with the ID 23199974236576.
master log message
`2026-09-29 19:09:58,840 INFO [.c.c.DefaultClassLoaderService]
[qtp1434741143-95] - Create classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:09:58,840 INFO [.c.c.DefaultClassLoaderService]
[qtp1434741143-95] - Create classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:09:58,979 INFO [.c.c.DefaultClassLoaderService]
[qtp1434741143-95] - Release classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:09:58,992 INFO [.c.c.DefaultClassLoaderService]
[qtp1434741143-95] - Release classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:09:59,036 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Create classloader for job 23199974236576
with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:09:59,036 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Create classloader for job 23199974236576
with jars []
2026-09-29 19:09:59,036 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Create classloader for job 23199974236576
with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:10:00,701 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Release classloader for job 23199974236576
with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:10:00,712 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Release classloader for job 23199974236576
with jars []
2026-09-29 19:10:00,720 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - Release classloader for job 23199974236576
with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:10:00,729 INFO [.c.c.DefaultClassLoaderService]
[seatunnel-coordinator-service-8] - recycle classloader for thread process
reaper`
worker log
`2026-09-29 19:10:01,173 INFO [.c.c.DefaultClassLoaderService]
[hz.main.generic-operation.thread-45] - Create classloader for job
23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:10:01,369 INFO [.c.c.DefaultClassLoaderService]
[hz.main.generic-operation.thread-47] - Create classloader for job
23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:23:59,169 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - Release classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-rtt-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.3:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.2:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread MaintenanceTimer-5-thread-1
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-rtt-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.4:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
hz.main.seaTunnel.task.thread-14
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread MaintenanceTimer-6-thread-1
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-rtt-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.2:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread BufferPoolPruner-1-thread-1
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.3:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
cluster-ClusterId{value='6abb9d0d9fa4f95b6f42db53',
description='null'}-172.30.41.4:27420
2026-09-29 19:23:59,173 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}
2026-09-29 19:23:59,174 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - Release classloader for job 23199974236576 with jars
[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar]
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread ResponseProcessor for block
BP-1405345564-172.30.41.51-1706777605594:blk_1119256020_45533959
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread DataStreamer for file
/tmp/seatunnel/seatunnel/23199974236576/77aaf92c3e/T_23199974236576_77aaf92c3e_0_1/pdate=2026-09-28/T_23199974236576_77aaf92c3e_0_1.snappy.parquet
block BP-1405345564-172.30.41.51-1706777605594:blk_1119256020_45533959
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
hz.main.seaTunnel.task.thread-15
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread LeaseRenewer:hueuser@ns-dev260
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - recycle classloader for thread
hz.main.seaTunnel.task.thread-11
2026-09-29 19:23:59,176 INFO [.c.c.DefaultClassLoaderService]
[BlockingWorker-TaskGroupLocation{jobId=23199974236576, pipelineId=1,
taskGroupId=3}] - Cleared 1 Hadoop ReflectionUtils cache entries for released
classloader`
<img width="2962" height="612" alt="Image"
src="https://github.com/user-attachments/assets/32de6f0c-21b0-42ba-8bf7-702074d0a8e4";
/>
<img width="2984" height="1332" alt="Image"
src="https://github.com/user-attachments/assets/50a2f6ff-9278-4e02-9084-da7ff797f23e";
/>
While the job was running, I could see that the classLoaderCache size was 1
using the following command:
[arthas@10]$ vmtool --action getInstances --className
org.apache.seatunnel.engine.core.classloader.DefaultClassLoaderService
--express 'instances[0].classLoaderCache.size()'
@Integer[1]
The details were as follows:
[arthas@10]$ vmtool --action getInstances --className
org.apache.seatunnel.engine.core.classloader.DefaultClassLoaderService
--express 'instances[0].classLoaderCache'
@ConcurrentHashMap[
@Long[23199974236576]:@ConcurrentHashMap[isEmpty=false;size=2],
]
[arthas@10]$
[arthas@10]$ vmtool --action getInstances --className
org.apache.seatunnel.engine.core.classloader.DefaultClassLoaderService
--express 'instances[0].classLoaderCache.values().iterator().next().keySet()'
@KeySetView[
@String[file:/opt/seatunnel/connectors/connector-mongodb-2.3.12.odeon-2.12.15.jar],
@String[file:/opt/seatunnel/connectors/connector-hive-2.3.12.odeon-2.12.15.jar],
]
[arthas@10]$
[arthas@10]$ vmtool --action getInstances --className
org.apache.seatunnel.engine.core.classloader.DefaultClassLoaderService
--express 'instances[0].classLoaderCache.values().iterator().next().values()'
@ValuesView[
@SeaTunnelChildFirstClassLoader[org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@54061fb7],
@SeaTunnelChildFirstClassLoader[org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@686faa75],
]
Then, after the job finished, we can see that the classLoaderCache size
successfully dropped back to 0:
[arthas@10]$ vmtool --action getInstances --className
org.apache.seatunnel.engine.core.classloader.DefaultClassLoaderService
--express 'instances[0].classLoaderCache.size()'
@Integer[0]
<img width="2938" height="1170" alt="Image"
src="https://github.com/user-attachments/assets/a1cbfb41-760b-43c9-b95e-f5d4eb04a41c";
/>
This confirms that, for this job, the entry in classLoaderCache was
successfully removed after the job completed.
mat
I also took a heap dump at this point, then analyzed it using MAT.
first
<img width="1422" height="790" alt="Image"
src="https://github.com/user-attachments/assets/6a75b842-92f0-43b0-b96b-223706ea0d8e";
/>
second
<img width="1325" height="411" alt="Image"
src="https://github.com/user-attachments/assets/98cb9383-2080-43b9-ab89-1a913d6a197e";
/>
and @SEZ9 After the job finishes, the numberOfInstances of
SeaTunnelChildFirstClassLoader does not decrease; it remains at 4. Can’t you
see the image in my comment above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
