Hi, One thing I found could be related is that my kafka queue is really low on volume, most of the time the pipeline is already catching up. I've tested writing the checkpoint/savepoint to the local filesystem, and it turns out it is also just an empty folder. Wondering have anyone seeing the same thing and how do I avoid this happening? Thanks!
Sincerely, Lydian Lee On Thu, Oct 13, 2022 at 1:35 PM Lydian <lydia...@gmail.com> wrote: > oops, the log seems not formatting well: > ``` > > 2022-10-05 17:05:32,027 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > Impulse -> [3]Reading message from > kafka/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaUnbounded/Read(KafkaUnboundedSource)/{ParDo(Out > 2022-10-05 17:19:34,272 INFO org.apache.flink.runtime.jobmaster.JobMaster > [] - Triggering cancel-with-savepoint for job > 1a68f74acf0ccf403693e2f228fa62a6. > 2022-10-05 17:19:34,285 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering > checkpoint 1 (type=SAVEPOINT) @ 1664990374275 for job > 1a68f74acf0ccf403693e2f228fa62a6. > 2022-10-05 17:19:34,287 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Making directory: > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > 2022-10-05 17:19:34,287 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - op_mkdirs += 1 -> 3 > 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - op_get_file_status += 1 -> 6 > 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Getting path status for > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > (flink/expansi > 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - S3GetFileStatus > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_metadata_requests += 1 -> 6 > 2022-10-05 17:19:34,381 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_list_requests += 1 -> 6 > 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Not Found: > s3a://affirm-stage-chrono/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - op_get_file_status += 1 -> 7 > 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Getting path status for > s3a://test-bucket/flink/expansion-service-example/savepoints > (flink/expansion-service-example/savepoints) > 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - S3GetFileStatus > s3a://test-bucket/flink/expansion-service-example/savepoints > 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_metadata_requests += 1 -> 7 > 2022-10-05 17:19:34,431 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_list_requests += 1 -> 7 > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Found path as directory (with /) > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Prefix count = 0; object count=1 > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Summary: flink/expansion-service-example/savepoints/ 0 > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT 0 bytes to > flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT start 0 bytes > 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_put_requests += 1 -> 3 > 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT completed success=true; 0 bytes > 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_put_requests_completed += 1 -> 3 > 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Finished write to > flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/, > len 0 > 2022-10-05 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem > [] - To delete unnecessary fake directory > flink/expansion-service-example/savepoints/ for > s3a://affirm-stage-chrono/flink/expansion-service-exa > 2022-10-05 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem > [] - To delete unnecessary fake directory > flink/expansion-service-example/ for > s3a://affirm-stage-chrono/flink/expansion-service-example > 2022-10-05 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem > [] - To delete unnecessary fake directory flink/ for > s3a://affirm-stage-chrono/flink > 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_delete_requests += 1 -> 3 > 2022-10-05 17:19:34,617 DEBUG > org.apache.hadoop.fs.s3a.S3AFileSystem.Progress [] - PUT > flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/: 0 > bytes > 2022-10-05 17:24:34,285 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint > 1 of job 1a68f74acf0ccf403693e2f228fa62a6 expired before completing. > 2022-10-05 17:24:34,287 WARN > org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to > trigger or complete checkpoint 1 for job 1a68f74acf0ccf403693e2f228fa62a6. (0 > consecutive failed attempts so far) > org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired > before completing. > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2000) > [flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_292] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_292] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_292] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_292] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_292] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] > 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - op_delete += 1 -> 1 > 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - op_get_file_status += 1 -> 8 > 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Getting path status for > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > (flink/expansi > 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - S3GetFileStatus > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_metadata_requests += 1 -> 8 > 2022-10-05 17:24:34,360 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_list_requests += 1 -> 8 > 2022-10-05 17:24:34,394 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Found path as directory (with /) > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Prefix count = 0; object count=1 > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Summary: > flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ 0 > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Delete path > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > - recursive true > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - delete: Path is a directory: > s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Deleting fake empty directory > flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ > 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_delete_requests += 1 -> 4 > 2022-10-05 17:24:34,427 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - S3GetFileStatus > s3a://test-bucket/flink/expansion-service-example/savepoints > 2022-10-05 17:24:34,427 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_metadata_requests += 1 -> 9 > 2022-10-05 17:24:34,444 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_list_requests += 1 -> 9 > 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Not Found: > s3a://test-bucket/flink/expansion-service-example/savepoints > 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Creating new fake directory at > s3a://test-bucket/flink/expansion-service-example/savepoints > 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT 0 bytes to flink/expansion-service-example/savepoints/ > 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT start 0 bytes > 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_put_requests += 1 -> 4 > 2022-10-05 17:24:34,548 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - PUT completed success=true; 0 bytes > 2022-10-05 17:24:34,548 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_put_requests_completed += 1 -> 4 > 2022-10-05 17:24:34,548 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem > [] - Finished write to > flink/expansion-service-example/savepoints/, len 0 > 2022-10-05 17:24:34,548 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem > [] - To delete unnecessary fake directory > flink/expansion-service-example/ for > s3a://affirm-stage-chrono/flink/expansion-service-example > 2022-10-05 17:24:34,548 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem > [] - To delete unnecessary fake directory flink/ for > s3a://affirm-stage-chrono/flink > 2022-10-05 17:24:34,548 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics > [] - object_delete_requests += 1 -> 5 > 2022-10-05 17:24:34,597 DEBUG > org.apache.hadoop.fs.s3a.S3AFileSystem.Progress [] - PUT > flink/expansion-service-example/savepoints/: 0 bytes > > ``` > Sincerely, > Lydian Lee > > > > On Thu, Oct 13, 2022 at 1:34 PM Lydian <lydia...@gmail.com> wrote: > >> Hi, >> >> I have a simple python pipeline running on top of flink runner, >> ``` >> pipeline | ReadFromKafka() | beam.Map(lambda args: logging.info(args[1])) >> ``` >> However, I found that it always put 0 checkpoint into s3, and there's no >> error log for why the checkpoint data length is 0. Wondering if anyone >> knows how I can fix this issue? Thanks! >> >> I am currently using flink 1.14.5 and beam 2.41.0, and here's the log: >> ``` >> 2022-10-05 17:05:32,027 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Impulse >> -> [3]Reading message from >> kafka/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaUnbounded/Read(KafkaUnboundedSource)/{ParDo(Out >> 2022-10-05 17:19:34,272 INFO org.apache.flink.runtime.jobmaster.JobMaster >> [] - Triggering cancel-with-savepoint for job >> 1a68f74acf0ccf403693e2f228fa62a6. 2022-10-05 17:19:34,285 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering >> checkpoint 1 (type=SAVEPOINT) @ 1664990374275 for job >> 1a68f74acf0ccf403693e2f228fa62a6. 2022-10-05 17:19:34,287 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Making directory: >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> 2022-10-05 17:19:34,287 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - op_mkdirs += 1 -> 3 2022-10-05 17:19:34,288 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - op_get_file_status += 1 >> -> 6 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Getting path status for >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> (flink/expansi 2022-10-05 17:19:34,288 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - S3GetFileStatus >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> 2022-10-05 17:19:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_metadata_requests += 1 -> 6 2022-10-05 17:19:34,381 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - object_list_requests += >> 1 -> 6 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Not Found: >> s3a://affirm-stage-chrono/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> 2022-10-05 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - op_get_file_status += 1 -> 7 2022-10-05 17:19:34,411 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Getting path status for >> s3a://test-bucket/flink/expansion-service-example/savepoints >> (flink/expansion-service-example/savepoints) 2022-10-05 17:19:34,411 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - S3GetFileStatus >> s3a://test-bucket/flink/expansion-service-example/savepoints 2022-10-05 >> 17:19:34,411 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - >> object_metadata_requests += 1 -> 7 2022-10-05 17:19:34,431 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - object_list_requests += >> 1 -> 7 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Found path as directory (with /) 2022-10-05 17:19:34,461 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Prefix count = 0; object >> count=1 2022-10-05 17:19:34,461 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Summary: >> flink/expansion-service-example/savepoints/ 0 2022-10-05 17:19:34,461 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - PUT 0 bytes to >> flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ >> 2022-10-05 17:19:34,461 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - >> PUT start 0 bytes 2022-10-05 17:19:34,461 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - object_put_requests += 1 >> -> 3 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - PUT completed success=true; 0 bytes 2022-10-05 17:19:34,527 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - >> object_put_requests_completed += 1 -> 3 2022-10-05 17:19:34,527 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Finished write to >> flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/, >> len 0 2022-10-05 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - To delete unnecessary fake directory >> flink/expansion-service-example/savepoints/ for >> s3a://affirm-stage-chrono/flink/expansion-service-exa 2022-10-05 >> 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem [] - To delete >> unnecessary fake directory flink/expansion-service-example/ for >> s3a://affirm-stage-chrono/flink/expansion-service-example 2022-10-05 >> 17:19:34,527 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem [] - To delete >> unnecessary fake directory flink/ for s3a://affirm-stage-chrono/flink >> 2022-10-05 17:19:34,527 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_delete_requests += 1 -> 3 2022-10-05 17:19:34,617 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem.Progress [] - PUT >> flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/: >> 0 bytes 2022-10-05 17:24:34,285 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint 1 >> of job 1a68f74acf0ccf403693e2f228fa62a6 expired before completing. >> 2022-10-05 17:24:34,287 WARN >> org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to >> trigger or complete checkpoint 1 for job 1a68f74acf0ccf403693e2f228fa62a6. >> (0 consecutive failed attempts so far) >> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired >> before completing. at >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2000) >> [flink-dist_2.12-1.14.5.jar:1.14.5] at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [?:1.8.0_292] at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [?:1.8.0_292] at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> [?:1.8.0_292] at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> [?:1.8.0_292] at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> [?:1.8.0_292] at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] >> 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - op_delete += 1 -> 1 2022-10-05 17:24:34,288 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - op_get_file_status += 1 >> -> 8 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Getting path status for >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> (flink/expansi 2022-10-05 17:24:34,288 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - S3GetFileStatus >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> 2022-10-05 17:24:34,288 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_metadata_requests += 1 -> 8 2022-10-05 17:24:34,360 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - object_list_requests += >> 1 -> 8 2022-10-05 17:24:34,394 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Found path as directory (with /) 2022-10-05 17:24:34,395 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Prefix count = 0; object >> count=1 2022-10-05 17:24:34,395 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Summary: >> flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ 0 >> 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - >> Delete path >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> - recursive true 2022-10-05 17:24:34,395 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - delete: Path is a directory: >> s3a://test-bucket/flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89 >> 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - >> Deleting fake empty directory >> flink/expansion-service-example/savepoints/savepoint-1a68f7-bb0ae207bd89/ >> 2022-10-05 17:24:34,395 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_delete_requests += 1 -> 4 2022-10-05 17:24:34,427 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - S3GetFileStatus >> s3a://test-bucket/flink/expansion-service-example/savepoints 2022-10-05 >> 17:24:34,427 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - >> object_metadata_requests += 1 -> 9 2022-10-05 17:24:34,444 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - object_list_requests += >> 1 -> 9 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem >> [] - Not Found: >> s3a://test-bucket/flink/expansion-service-example/savepoints 2022-10-05 >> 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - Creating new >> fake directory at >> s3a://test-bucket/flink/expansion-service-example/savepoints 2022-10-05 >> 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - PUT 0 bytes >> to flink/expansion-service-example/savepoints/ 2022-10-05 17:24:34,482 >> DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - PUT start 0 bytes >> 2022-10-05 17:24:34,482 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_put_requests += 1 -> 4 2022-10-05 17:24:34,548 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - PUT completed success=true; 0 >> bytes 2022-10-05 17:24:34,548 DEBUG >> org.apache.hadoop.fs.s3a.S3AStorageStatistics [] - >> object_put_requests_completed += 1 -> 4 2022-10-05 17:24:34,548 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem [] - Finished write to >> flink/expansion-service-example/savepoints/, len 0 2022-10-05 17:24:34,548 >> TRACE org.apache.hadoop.fs.s3a.S3AFileSystem [] - To delete unnecessary >> fake directory flink/expansion-service-example/ for >> s3a://affirm-stage-chrono/flink/expansion-service-example 2022-10-05 >> 17:24:34,548 TRACE org.apache.hadoop.fs.s3a.S3AFileSystem [] - To delete >> unnecessary fake directory flink/ for s3a://affirm-stage-chrono/flink >> 2022-10-05 17:24:34,548 DEBUG org.apache.hadoop.fs.s3a.S3AStorageStatistics >> [] - object_delete_requests += 1 -> 5 2022-10-05 17:24:34,597 DEBUG >> org.apache.hadoop.fs.s3a.S3AFileSystem.Progress [] - PUT >> flink/expansion-service-example/savepoints/: 0 bytes >> ``` >> >> >> >> Sincerely, >> Lydian Lee >> >>