YuriyGavrilov opened a new issue, #7650: URL: https://github.com/apache/seatunnel/issues/7650
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Long load csv sample data to iceberg s3 after few hours get an error but there are some files on s3 generated a bit more than 6000 (base) yuriygavrilov@MBP-Yuriy mvn % mc ls storj/test/iceberg_p_listner5_podman/taxi3-65b2d02895a249789ee17c62fce0b4e3/data | head [2024-09-11 22:45:05 MSK] 86KiB STANDARD 00001-1726083421402-b1287377-2b23-48d9-9671-47aaae671947-00319.parquet [2024-09-12 00:50:05 MSK] 93KiB STANDARD 00001-1726083421402-b1287377-2b23-48d9-9671-47aaae671947-05217.parquet [2024-09-12 00:25:30 MSK] 89KiB STANDARD 00001-1726083421402-b1287377-2b23-48d9-9671-47aaae671947-04249.parquet [2024-09-12 00:44:39 MSK] 90KiB STANDARD 00001-1726083421402-b1287377-2b23-48d9-9671-47aaae671947-05003.parquet Then i surprised that there are a lot of small files so i changed config and repeat the same job with 100mb files local file row count is: 112 234 627 second job statistics: 2024-09-12 18:41:02,508 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-886677532135391233] - *********************************************** Job Progress Information *********************************************** Job Id : 886677532135391233 Read Count So Far : 110820371 Write Count So Far : 110820356 Average Read Count : 88386/s Average Write Count : 88386/s Last Statistic Time : 2024-09-12 18:40:02 Current Statistic Time : 2024-09-12 18:41:02 *********************************************** First error reason is: full two logs below ``` 2024-09-12 18:45:10,132 ERROR [o.a.s.e.c.j.ClientJobProxy ] [main] - Job (886677532135391233) end with unknown state, and throw Exception: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=12, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 258a26f6-a561-4fcb-9e85-c94f0b7af724, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-12 18:45:10.081. Start time: 2024-09-12 18:20:02.492. Total elapsed time: 1507589 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobCompleteV2$0(ClientJobProxy.java:100) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobCompleteV2(ClientJobProxy.java:96) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:201) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=12, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 258a26f6-a561-4fcb-9e85-c94f0b7af724, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-12 18:45:10.081. Start time: 2024-09-12 18:20:02.492. Total elapsed time: 1507589 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:41:52.177, lastWriteTime=2024-09-12 18:41:52.176, closedTime=never, connected server version=5.1} ... 10 more ``` ### SeaTunnel Version latest 2.3.8 manually build two days ago ### SeaTunnel Config ```conf # Set the basic configuration of the task to be performed env { parallelism = 7 job.mode = "batch" # checkpoint.interval = 30000 # checkpoint.timeout = 5000 } # Create a source to connect to Clickhouse source { LocalFile { schema { fields { VendorID = string tpep_pickup_datetime = string tpep_dropoff_datetime = string passenger_count = string trip_distance = string RatecodeID = string store_and_fwd_flag = string PULocationID = string DOLocationID = string payment_type = string fare_amount = string extra = string mta_tax = string tip_amount = string tolls_amount = string improvement_surcharge = string total_amount = string } } path = "./2018_Yellow_Taxi_Trip_Data.csv" file_format_type = "csv" field_delimiter = "," # datetime_format = "dd/MM/yyyy hh:mm:ss" skip_header_row_number = 1 } } transform { } # Console printing of the read Clickhouse data sink { iceberg { catalog_name = "iceberg" iceberg.catalog.config={ "type"="hive" "uri" = "thrift://metastore:9083" "warehouse"="s3a://test/iceberg_p_listner5_podman/" } hadoop.config={ "fs.s3a.aws.credentials.provider" = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" "fs.s3a.endpoint" = "gateway.storjshare.io" "fs.s3a.access.key" = "jvrXXXXXXXXXyjrbq" "fs.s3a.secret.key" = "jzwnXXXXXXXXXXXXXmhhs" "fs.defaultFS" = "s3a://test/iceberg_p_listner5_podman/" "fs.s3a.impl"="org.apache.hadoop.fs.s3a.S3AFileSystem" } namespace = "my_schema_i" table = "taxi3" iceberg.table.write-props={ write.format.default="parquet" write.parquet.compression-codec="snappy" write.target-file-size-bytes=536870 } # iceberg.table.primary-keys="id" # iceberg.table.upsert-mode-enabled=true # iceberg.table.schema-evolution-enabled=true # case_sensitive=true # result_table_name = "test_table" } } ``` ### Running Command ```shell ./bin/seatunnel.sh --config ./config/V2.LLM.csv-ice1.config.template -m local ``` ### Error Exception ```log 2024-09-11 22:29:00,059 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-886334518498164737] - *********************************************** Job Progress Information *********************************************** Job Id : 886334518498164737 Read Count So Far : 32913074 Write Count So Far : 32911024 Average Read Count : 3135/s Average Write Count : 3135/s Last Statistic Time : 2024-09-11 22:28:00 Current Statistic Time : 2024-09-11 22:29:00 *********************************************** 2024-09-11 22:59:10,779 WARN [c.h.s.i.o.i.InvocationMonitor ] [hz.main.InvocationMonitorThread] - [localhost]:5801 [seatunnel-151894] [5.1] MonitorInvocationsTask delayed 1800758 ms 2024-09-11 22:59:10,779 WARN [c.h.s.i.o.i.InvocationMonitor ] [hz.main.InvocationMonitorThread] - [localhost]:5801 [seatunnel-151894] [5.1] BroadcastOperationControlTask delayed 1800759 ms 2024-09-11 22:59:11,779 INFO [.i.c.i.ClusterHeartbeatManager] [hz.main.cached.thread-10] - [localhost]:5801 [seatunnel-151894] [5.1] System clock apparently jumped from 2024-09-11 22:29:09.015 to 2024-09-11 22:59:11.773 since last heartbeat (+1800758 ms) 2024-09-11 22:59:11,779 WARN [.i.c.i.ClusterHeartbeatManager] [hz.main.cached.thread-10] - [localhost]:5801 [seatunnel-151894] [5.1] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 1800758 ms, Heartbeat-Timeout: 180000 ms 2024-09-11 22:59:12,013 WARN [c.h.c.i.c.t.HeartbeatManager ] [hz.client_1.internal-1] - hz.client_1 [seatunnel-151894] [5.1] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51863->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-11 22:29:06.336, lastWriteTime=2024-09-11 22:29:06.336, closedTime=never, connected server version=5.1} 2024-09-11 22:59:12,014 WARN [.h.c.i.c.t.TcpClientConnection] [hz.client_1.internal-1] - hz.client_1 [seatunnel-151894] [5.1] ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51863->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-11 22:29:06.336, lastWriteTime=2024-09-11 22:29:06.336, closedTime=2024-09-11 22:59:12.013, connected server version=5.1} closed. Reason: Heartbeat timed out com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51863->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-11 22:29:06.336, lastWriteTime=2024-09-11 22:29:06.336, closedTime=never, connected server version=5.1} at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.8-SNAPSHOT] 2024-09-11 22:59:12,027 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-151894] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51863, qualifier=null, endpoint=[127.0.0.1]:51863, remoteUuid=c9683c8f-7dfa-4987-9fd5-5bf32f9223ab, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-09-11 22:59:12,032 INFO [c.h.c.i.ClientEndpointManager ] [hz.main.event-4] - [localhost]:5801 [seatunnel-151894] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51863, qualifier=null, endpoint=[127.0.0.1]:51863, remoteUuid=c9683c8f-7dfa-4987-9fd5-5bf32f9223ab, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=c9683c8f-7dfa-4987-9fd5-5bf32f9223ab, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1726083416272, latest clientAttributes=lastStatisticsCollectionTime=1726093746333,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1726083416259,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=5436399616,os.freePhysicalMemorySize=3977764864,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=1048576,os.openFileDescriptorCount=81,os.processCpuTime=1003080000000,os.systemLoadAverage=0.5029296875,os.totalPhysicalMemorySize=155965644 80,os.totalSwapSpaceSize=0,runtime.availableProcessors=6,runtime.freeMemory=170950240,runtime.maxMemory=529530880,runtime.totalMemory=529530880,runtime.uptime=10333477,runtime.usedMemory=358582960, labels=[]} 2024-09-11 22:59:12,032 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-1] - hz.client_1 [seatunnel-151894] [5.1] Removed connection to endpoint: [localhost]:5801:61123c25-8b01-4585-b5cc-b20a740b02cc, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51863->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-11 22:29:06.336, lastWriteTime=2024-09-11 22:29:06.336, closedTime=2024-09-11 22:59:12.013, connected server version=5.1} 2024-09-11 22:59:12,032 INFO [c.h.c.LifecycleService ] [hz.client_1.internal-1] - hz.client_1 [seatunnel-151894] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2024-09-11 22:59:12,033 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-2] - hz.client_1 [seatunnel-151894] [5.1] Trying to connect to cluster: seatunnel-151894 2024-09-11 22:59:12,041 ERROR [o.a.s.e.c.j.ClientJobProxy ] [main] - Job (886334518498164737) end with unknown state, and throw Exception: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=11, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 61123c25-8b01-4585-b5cc-b20a740b02cc, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-11 22:59:12.028. Start time: 2024-09-11 19:37:00.048. Total elapsed time: 12131980 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobCompleteV2$0(ClientJobProxy.java:100) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobCompleteV2(ClientJobProxy.java:96) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:201) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=11, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 61123c25-8b01-4585-b5cc-b20a740b02cc, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-11 22:59:12.028. Start time: 2024-09-11 19:37:00.048. Total elapsed time: 12131980 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51863->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-11 22:29:06.336, lastWriteTime=2024-09-11 22:29:06.336, closedTime=never, connected server version=5.1} ... 10 more 2024-09-11 22:59:12,041 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-2] - hz.client_1 [seatunnel-151894] [5.1] Trying to connect to Member [localhost]:5801 - 61123c25-8b01-4585-b5cc-b20a740b02cc 2024-09-11 22:59:12,043 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-151894] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2024-09-11 22:59:12,057 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-2] - [localhost]:5801 [seatunnel-151894] [5.1] Connection[id=5, /127.0.0.1:5801->/127.0.0.1:37983, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-09-11 22:59:12,057 INFO [.p.t.AuthenticationMessageTask] [hz.main.priority-generic-operation.thread-0] - [localhost]:5801 [seatunnel-151894] [5.1] Received auth from Connection[id=5, /127.0.0.1:5801->/127.0.0.1:37983, qualifier=null, endpoint=[127.0.0.1]:37983, remoteUuid=null, alive=false, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: c9683c8f-7dfa-4987-9fd5-5bf32f9223ab, client name: hz.client_1, client version: 5.1 2024-09-11 22:59:12,058 WARN [.c.i.s.ClientInvocationService] [hz.client_1.internal-2] - hz.client_1 [seatunnel-151894] [5.1] Future.complete(Object) on completed future. Request: ClientMessage{connection=null, length=121, operation=Client.Authentication, isRetryable=true, correlationId=2253, messageType=100, isEvent=false, isfragmented=false}, current value: ExceptionalResult{cause=java.lang.InterruptedException}, offered value: ExceptionalResult{cause=com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down} some other logs lines 2024-09-11 22:59:13,374 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - Stopping s3a-file-system metrics system... 2024-09-11 22:59:13,375 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - s3a-file-system metrics system stopped. 2024-09-11 22:59:13,376 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - s3a-file-system metrics system shutdown complete. ----------- second run: ----------- 2024-09-12 18:41:02,508 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-886677532135391233] - *********************************************** Job Progress Information *********************************************** Job Id : 886677532135391233 Read Count So Far : 110820371 Write Count So Far : 110820356 Average Read Count : 88386/s Average Write Count : 88386/s Last Statistic Time : 2024-09-12 18:40:02 Current Statistic Time : 2024-09-12 18:41:02 *********************************************** 2024-09-12 18:45:07,686 WARN [c.h.s.i.o.i.InvocationMonitor ] [hz.main.InvocationMonitorThread] - [localhost]:5801 [seatunnel-638407] [5.1] MonitorInvocationsTask delayed 192947 ms 2024-09-12 18:45:08,610 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-638407] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 33 taskCount : 34 *********************************************** 2024-09-12 18:45:08,660 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-638407] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 *********************************************** 2024-09-12 18:45:08,674 WARN [c.h.s.i.o.i.InvocationMonitor ] [hz.main.InvocationMonitorThread] - [localhost]:5801 [seatunnel-638407] [5.1] BroadcastOperationControlTask delayed 192953 ms 2024-09-12 18:45:08,771 INFO [.i.c.i.ClusterHeartbeatManager] [hz.main.cached.thread-4] - [localhost]:5801 [seatunnel-638407] [5.1] System clock apparently jumped from 2024-09-12 18:41:53.699 to 2024-09-12 18:45:08.647 since last heartbeat (+192948 ms) 2024-09-12 18:45:08,771 WARN [.i.c.i.ClusterHeartbeatManager] [hz.main.cached.thread-4] - [localhost]:5801 [seatunnel-638407] [5.1] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 192948 ms, Heartbeat-Timeout: 180000 ms 2024-09-12 18:45:10,004 WARN [c.h.c.i.c.t.HeartbeatManager ] [hz.client_1.internal-1] - hz.client_1 [seatunnel-638407] [5.1] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:41:52.177, lastWriteTime=2024-09-12 18:41:52.176, closedTime=never, connected server version=5.1} 2024-09-12 18:45:10,014 WARN [.h.c.i.c.t.TcpClientConnection] [hz.client_1.internal-1] - hz.client_1 [seatunnel-638407] [5.1] ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:41:52.177, lastWriteTime=2024-09-12 18:41:52.176, closedTime=2024-09-12 18:45:10.005, connected server version=5.1} closed. Reason: Heartbeat timed out com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:41:52.177, lastWriteTime=2024-09-12 18:41:52.176, closedTime=never, connected server version=5.1} at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.8-SNAPSHOT] 2024-09-12 18:45:10,066 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-638407] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51187, qualifier=null, endpoint=[127.0.0.1]:51187, remoteUuid=961350bc-635e-454f-bf38-c21eac4fa1a9, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-09-12 18:45:10,090 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-1] - hz.client_1 [seatunnel-638407] [5.1] Removed connection to endpoint: [localhost]:5801:258a26f6-a561-4fcb-9e85-c94f0b7af724, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:45:10.057, lastWriteTime=2024-09-12 18:41:52.176, closedTime=2024-09-12 18:45:10.005, connected server version=5.1} 2024-09-12 18:45:10,090 INFO [c.h.c.LifecycleService ] [hz.client_1.internal-1] - hz.client_1 [seatunnel-638407] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2024-09-12 18:45:10,092 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-8] - hz.client_1 [seatunnel-638407] [5.1] Trying to connect to cluster: seatunnel-638407 2024-09-12 18:45:10,093 INFO [.c.i.c.ClientConnectionManager] [hz.client_1.internal-8] - hz.client_1 [seatunnel-638407] [5.1] Trying to connect to Member [localhost]:5801 - 258a26f6-a561-4fcb-9e85-c94f0b7af724 2024-09-12 18:45:10,132 ERROR [o.a.s.e.c.j.ClientJobProxy ] [main] - Job (886677532135391233) end with unknown state, and throw Exception: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=12, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 258a26f6-a561-4fcb-9e85-c94f0b7af724, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-12 18:45:10.081. Start time: 2024-09-12 18:20:02.492. Total elapsed time: 1507589 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobCompleteV2$0(ClientJobProxy.java:100) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobCompleteV2(ClientJobProxy.java:96) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:201) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=12, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 258a26f6-a561-4fcb-9e85-c94f0b7af724, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-09-12 18:45:10.081. Start time: 2024-09-12 18:20:02.492. Total elapsed time: 1507589 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:51187->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-09-12 18:41:52.177, lastWriteTime=2024-09-12 18:41:52.176, closedTime=never, connected server version=5.1} ... 10 more 2024-09-12 18:45:10,138 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-638407] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2024-09-12 18:45:10,142 INFO [c.h.c.i.ClientEndpointManager ] [hz.main.event-3] - [localhost]:5801 [seatunnel-638407] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51187, qualifier=null, endpoint=[127.0.0.1]:51187, remoteUuid=961350bc-635e-454f-bf38-c21eac4fa1a9, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=961350bc-635e-454f-bf38-c21eac4fa1a9, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1726165197076, latest clientAttributes=lastStatisticsCollectionTime=1726166512174,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1726165197049,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=5411364864,os.freePhysicalMemorySize=6940622848,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=1048576,os.openFileDescriptorCount=83,os.processCpuTime=2704820000000,os.systemLoadAverage=2.587890625,os.totalPhysicalMemorySize=1559656448 0,os.totalSwapSpaceSize=0,runtime.availableProcessors=6,runtime.freeMemory=271620976,runtime.maxMemory=532676608,runtime.totalMemory=532676608,runtime.uptime=1319575,runtime.usedMemory=263890096, labels=[]} 2024-09-12 18:45:10,147 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-638407] [5.1] Connection[id=2, /127.0.0.1:5801->/127.0.0.1:46933, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-09-12 18:45:10,158 INFO [.p.t.AuthenticationMessageTask] [hz.main.generic-operation.thread-12] - [localhost]:5801 [seatunnel-638407] [5.1] Received auth from Connection[id=2, /127.0.0.1:5801->/127.0.0.1:46933, qualifier=null, endpoint=[127.0.0.1]:46933, remoteUuid=null, alive=false, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 961350bc-635e-454f-bf38-c21eac4fa1a9, client name: hz.client_1, client version: 5.1 2024-09-12 18:45:10,154 WARN [.c.i.s.ClientInvocationService] [hz.client_1.internal-8] - hz.client_1 [seatunnel-638407] [5.1] Future.complete(Object) on completed future. Request: ClientMessage{connection=null, length=121, operation=Client.Authentication, isRetryable=true, correlationId=298, messageType=100, isEvent=false, isfragmented=false}, current value: ExceptionalResult{cause=java.lang.InterruptedException}, offered value: ExceptionalResult{cause=com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down} java.lang.Exception: null at com.hazelcast.spi.impl.AbstractInvocationFuture.warnIfSuspiciousDoubleCompletion(AbstractInvocationFuture.java:1253) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1230) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:316) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.authenticateOnCluster(TcpClientConnectionManager.java:885) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToMember(TcpClientConnectionManager.java:627) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$2(TcpClientConnectionManager.java:492) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:492) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$submitConnectToClusterTask$1(TcpClientConnectionManager.java:385) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.8-SNAPSHOT] some other logs lines java.util.concurrent.ExecutionException: com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_422] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_422] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$execute$1(ClientExecuteCommand.java:181) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] Caused by: com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:316) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:209) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invoke(ClientInvocation.java:156) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestAndGetCompletableFuture(SeaTunnelHazelcastClient.java:115) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestOnMasterAndGetCompletableFuture(SeaTunnelHazelcastClient.java:124) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.job.ClientJobProxy.cancelJob(ClientJobProxy.java:142) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.shutdownHook(ClientExecuteCommand.java:281) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$execute$0(ClientExecuteCommand.java:178) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_422] at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) ~[?:1.8.0_422] Caused by: java.io.IOException: No connection found to cluster. at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkInvocationAllowed(TcpClientConnectionManager.java:548) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.checkInvocationAllowed(ClientInvocationServiceImpl.java:294) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:178) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invoke(ClientInvocation.java:156) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestAndGetCompletableFuture(SeaTunnelHazelcastClient.java:115) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestOnMasterAndGetCompletableFuture(SeaTunnelHazelcastClient.java:124) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.engine.client.job.ClientJobProxy.cancelJob(ClientJobProxy.java:142) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.shutdownHook(ClientExecuteCommand.java:281) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$execute$0(ClientExecuteCommand.java:178) ~[seatunnel-starter.jar:2.3.8-SNAPSHOT] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_422] at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_422] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) ~[?:1.8.0_422] 2024-09-12 18:45:14,145 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - Stopping s3a-file-system metrics system... 2024-09-12 18:45:14,146 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - s3a-file-system metrics system stopped. 2024-09-12 18:45:14,147 INFO [o.a.h.m.i.MetricsSystemImpl ] [shutdown-hook-0] - s3a-file-system metrics system shutdown complete. ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots Trino select try: nothing to show <img width="533" alt="Снимок экрана 2024-09-12 в 21 52 12" src="https://github.com/user-attachments/assets/556733f7-7e80-4957-af98-f572173cef29";> ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
