Is the 4th "checkpointed size" and "checkpoint duration" bigger than others? If it is true, I guess it's related to the flush of rocksdb. It may delay the next checkpoint.
Best, Hangxiang. On Fri, Jun 17, 2022 at 2:31 PM Hangxiang Yu <master...@gmail.com> wrote: > Is the 4th "checkpointed size" and "checkpoint duration" bigger than > others? If it is true, I guess it's related to the flush of rocksdb. > It may delay the next checkpoint. > > Best, > Hangxiang. > > On Thu, Jun 16, 2022 at 3:12 AM Jai Patel <jai.pa...@cloudkitchens.com> > wrote: > >> I was thinking I could check the metrics to see if a flush is happening. >> Which metrics (RocksDB related or not) should I look at? I checked >> busy/backpressure but these seemed to be spiking after the checkpoint was >> complete (catching up on the backlog of messages accumulated). >> >> Thanks. >> Jai >> >> On Wed, Jun 15, 2022 at 11:35 AM Jai Patel <jai.pa...@cloudkitchens.com> >> wrote: >> >>> Thank you Hangxiang, >>> >>> Here are the details you asked about. >>> >>> Thanks. >>> Jai >>> >>> executionEnvironment.configure(configuration, >>> getClass().getClassLoader()); >>> >>> executionEnvironment.getConfig().setGlobalJobParameters(configuration); >>> executionEnvironment.getConfig().enableObjectReuse(); >>> >>> executionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); >>> >>> // Configure Checkpoints >>> CheckpointConfig checkpointConfig = >>> executionEnvironment.getCheckpointConfig(); >>> >>> checkpointConfig.setCheckpointTimeout(Duration.ofMinutes(40).toMillis()); >>> >>> checkpointConfig.setCheckpointInterval(Duration.ofMinutes(3).toMillis()); >>> >>> checkpointConfig.setMinPauseBetweenCheckpoints(Duration.ofSeconds(5).toMillis()); >>> >>> >>> Checkpoint Detail: >>> Path: >>> >>> gs://css-data-warehouse-flink/analytics-pipeline-app/checkpoints/5539f7042ecfd43bdeef7b102e03e964/chk-108588 >>> Discarded: - Checkpoint Type: aligned checkpoint >>> Operators: >>> NameAcknowledgedLatest AcknowledgmentEnd to End DurationCheckpointed >>> Data SizeProcessed (persisted) in-flight data >>> >>> Source: Kafka OrderChangeEvents >>> 12/12 (100%) 2022-06-15 11:30:04 2m 19s 56.9 KB 0 B (0 B) >>> >>> SubTasks: >>> End to End DurationCheckpointed Data SizeSync DurationAsync >>> DurationProcessed >>> (persisted) DataAlignment DurationStart Delay >>> Minimum 2m 19s 4.74 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s >>> Average 2m 19s 4.74 KB 0ms 6ms 0 B (0 B) 0ms 2m 19s >>> Maximum 2m 19s 4.74 KB 0ms 27ms 0 B (0 B) 0ms 2m 19s >>> IDAcknowledgedEnd to End DurationCheckpointed Data SizeSync DurationAsync >>> DurationProcessed (persisted) DataAlignment DurationStart DelayUnaligned >>> Checkpoint >>> 0 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 1 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 14ms 0 B (0 B) 0ms 2m 19s false >>> 2 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 3ms 0 B (0 B) 0ms 2m 19s false >>> 3 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 2ms 0 B (0 B) 0ms 2m 19s false >>> 4 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 27ms 0 B (0 B) 0ms 2m 19s false >>> 5 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 11ms 0 B (0 B) 0ms 2m 19s false >>> 6 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 2ms 0 B (0 B) 0ms 2m 19s false >>> 7 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 8 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 10ms 0 B (0 B) 0ms 2m 19s false >>> 9 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 2ms 0 B (0 B) 0ms 2m 19s false >>> 10 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 11 2022-06-15 11:30:04 2m 19s 4.74 KB 0ms 9ms 0 B (0 B) 0ms 2m 19s false >>> Source: Kafka ImportedTransactions >>> 18/18 (100%) 2022-06-15 11:30:04 2m 19s 85.1 KB 0 B (0 B) >>> >>> SubTasks: >>> End to End DurationCheckpointed Data SizeSync DurationAsync >>> DurationProcessed >>> (persisted) DataAlignment DurationStart Delay >>> Minimum 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s >>> Average 2m 19s 4.73 KB 0ms 5ms 0 B (0 B) 0ms 2m 19s >>> Maximum 2m 19s 4.73 KB 0ms 20ms 0 B (0 B) 0ms 2m 19s >>> IDAcknowledgedEnd to End DurationCheckpointed Data SizeSync DurationAsync >>> DurationProcessed (persisted) DataAlignment DurationStart DelayUnaligned >>> Checkpoint >>> 0 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 1 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 2ms 0 B (0 B) 0ms 2m 19s false >>> 2 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 10ms 0 B (0 B) 0ms 2m 19s false >>> 3 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 9ms 0 B (0 B) 0ms 2m 19s false >>> 4 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 14ms 0 B (0 B) 0ms 2m 19s false >>> 5 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 20ms 0 B (0 B) 0ms 2m 19s false >>> 6 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 7 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 8 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 12ms 0 B (0 B) 0ms 2m 19s false >>> 9 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 10 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 11 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 9ms 0 B (0 B) 0ms 2m 19s false >>> 12 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 8ms 0 B (0 B) 0ms 2m 19s false >>> 13 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 14 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 2ms 0 B (0 B) 0ms 2m 19s false >>> 15 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 16 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 1ms 0 B (0 B) 0ms 2m 19s false >>> 17 2022-06-15 11:30:04 2m 19s 4.73 KB 0ms 3ms 0 B (0 B) 0ms 2m 19s false >>> Source: Kafka Order Reenrichment EnrichedOrder -> Map >>> 12/12 (100%) 2022-06-15 11:30:04 2m 19s 57.0 KB 0 B (0 B) >>> >>> SubTasks: >>> End to End DurationCheckpointed Data SizeSync DurationAsync >>> DurationProcessed >>> (persisted) DataAlignment DurationStart Delay >>> Minimum 2m 19s 4.75 KB 0ms 0ms 0 B (0 B) 0ms 2m 19s >>> Average 2m 19s 4.75 KB >>> >>
