github-actions[bot] commented on code in PR #65219:
URL: https://github.com/apache/doris/pull/65219#discussion_r3548918942
##########
fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java:
##########
@@ -510,13 +581,36 @@ private static String convertToUtcTimestamp(Column
column, String border, TimeZo
return border;
}
+ /**
+ * For TIMESTAMPTZ partition columns, convert a UTC datetime string
+ * (e.g. "2027-01-01 00:00:00") to wall-clock time in
+ * {@link TimeUtils#getTimeZone()}. PropertyAnalyzer.analyzeDataProperty()
+ * parses lifecycle values as DATETIME via
DateLiteralUtils.createDateLiteral(),
+ * which treats a suffix-less string as local time in the session/JVM
+ * timezone and applies the current Instant's offset during the initial
+ * shift while using the target date's offset in unixTimestamp().
Converting
+ * to wall-clock time here avoids the DST-crossing offset mismatch that
+ * would otherwise shift the cooldown by one hour.
+ */
+ private static String convertUtcToSessionWallClock(Column column, String
utcDateTimeStr) {
+ if (column.getDataType() == PrimitiveType.TIMESTAMPTZ) {
+ ZonedDateTime local = LocalDateTime
+ .parse(utcDateTimeStr,
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))
+ .atZone(ZoneOffset.UTC)
+ .withZoneSameInstant(TimeUtils.getTimeZone().toZoneId());
+ return DateTimeFormatter.ofPattern("yyyy-MM-dd
HH:mm:ss").format(local);
+ }
Review Comment:
This still loses information for DST fall-back hours. For an hourly
TIMESTAMPTZ dynamic partition, a real UTC cooldown boundary such as `2026-11-01
07:00:00Z` in `America/Chicago` is converted here to the suffix-less wall time
`2026-11-01 01:00:00`. `PropertyAnalyzer.analyzeDataProperty()` later parses
that as a DATETIME in the same timezone, and Java resolves the ambiguous local
`01:00` to the earlier offset (`2026-11-01T06:00:00Z`), so the partition cools
down an hour before the intended boundary. The two UTC instants in the overlap
both collapse to the same string, so this needs an instant-preserving
representation/parser rather than dropping the offset.
##########
fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java:
##########
@@ -2313,4 +2514,595 @@ public void
testAutoPartitionRetentionTimestampTzCutoffNormalized() throws Excep
connectContext.getSessionVariable().setTimeZone(originalSessionTz);
}
}
+
+ @Test
+ public void testTimeStampTzGetHistoricalPartitionsRangeBased() throws
Exception {
+ // Verify that getHistoricalPartitions correctly identifies the current
+ // partition by range lower bound (currentUtcBorder) rather than by
name.
+ // When the configured timezone (e.g. Asia/Shanghai) is a day ahead of
UTC,
+ // the scheduler computes nowPartitionName from the TZ date (e.g.
"p20260707")
+ // while currentUtcBorder is from the UTC date ("2026-07-06
00:00:00+00:00").
+ // Name-based comparison would fail to exclude the real current
partition
+ // (p20260706), treating it as historical. Range-based comparison
correctly
+ // excludes it by matching the stored partition lower bound.
+
+ String createSql = "CREATE TABLE test.`tstz_hist_parts` (\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"true\",\n"
+ + "\"dynamic_partition.time_unit\" = \"day\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" = \"Asia/Shanghai\"\n"
+ + ");";
+ createTable(createSql);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("tstz_hist_parts");
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(), table.getId());
+
+ int totalPartitions = table.getPartitionNames().size();
+ Assert.assertEquals(7, totalPartitions);
+
+ RangePartitionInfo info = (RangePartitionInfo)
table.getPartitionInfo();
+ DynamicPartitionProperty prop =
table.getTableProperty().getDynamicPartitionProperty();
+ String partitionFormat = DynamicPartitionUtil.getPartitionFormat(
+ info.getPartitionColumns().get(0));
+ ZonedDateTime utcNow = ZonedDateTime.now(ZoneOffset.UTC);
+ // getPartitionRangeString with DATETIME_FORMAT returns "yyyy-MM-dd
HH:mm:ss"
+ // (e.g. "2026-07-06 00:00:00"). convertToUtcTimestamp appends
"+00:00".
+ String utcBorderDateTime =
DynamicPartitionUtil.getPartitionRangeString(prop, utcNow, 0, partitionFormat);
+ String currentUtcBorder = utcBorderDateTime + "+00:00";
+
+ // Use a deliberately mismatched nowPartitionName to simulate the
scenario
+ // where the configured TZ and UTC disagree on the calendar date.
+ // Name-based comparison would NOT exclude any partition with this
name.
+ String wrongNowPartitionName = "p_nonexistent";
+
+ // 1. Without currentUtcBorder: name-based cannot identify the current
partition.
+ List<Partition> historicalNoUtc =
DynamicPartitionScheduler.getHistoricalPartitions(
+ table, wrongNowPartitionName, null);
+ boolean currentFoundByName = false;
+ for (Partition p : historicalNoUtc) {
+ RangePartitionItem item = (RangePartitionItem)
info.getItem(p.getId());
+ String lowerBound =
item.getItems().lowerEndpoint().getKeys().get(0).getStringValue();
+ if (lowerBound.equals(currentUtcBorder)) {
+ currentFoundByName = true;
+ break;
+ }
+ }
+ Assert.assertTrue("Name-based should fail to exclude current
partition: "
+ + "nowPartitionName='" + wrongNowPartitionName + "' does not
match any partition",
+ currentFoundByName);
+
+ // 2. With currentUtcBorder: range-based correctly excludes the
current partition.
+ List<Partition> historicalWithUtc =
DynamicPartitionScheduler.getHistoricalPartitions(
+ table, wrongNowPartitionName, currentUtcBorder);
+ boolean currentFoundByRange = false;
+ for (Partition p : historicalWithUtc) {
+ RangePartitionItem item = (RangePartitionItem)
info.getItem(p.getId());
+ String lowerBound =
item.getItems().lowerEndpoint().getKeys().get(0).getStringValue();
+ if (lowerBound.equals(currentUtcBorder)) {
+ currentFoundByRange = true;
+ break;
+ }
+ }
+ Assert.assertFalse("Range-based should correctly exclude the current
partition",
+ currentFoundByRange);
+ Assert.assertEquals("Should exclude exactly the current partition",
+ totalPartitions - 1, historicalWithUtc.size());
+ }
+
+ @Test
+ public void testTimeStampTzDynamicPartitionMonthUnit() throws Exception {
+ // Verify TIMESTAMPTZ dynamic partition with MONTH unit:
+ // boundaries are at UTC midnight (day=01 00:00:00+00:00),
+ // names use the configured timezone's calendar month.
+ String originalTimeZone =
connectContext.getSessionVariable().getTimeZone();
+ try {
+ connectContext.getSessionVariable().setTimeZone("America/Chicago");
+
+ String createOlapTblStmt = "CREATE TABLE
test.`timestamptz_dynamic_month` (\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"true\",\n"
+ + "\"dynamic_partition.time_unit\" = \"month\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" = \"Asia/Shanghai\"\n"
+ + ");";
+ createTable(createOlapTblStmt);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("timestamptz_dynamic_month");
+ Assert.assertTrue(table.dynamicPartitionExists());
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(),
table.getId());
+
+ int partitionCount = table.getPartitionNames().size();
+ Assert.assertEquals(7, partitionCount);
+
+ // Verify partition boundaries are UTC midnight and names use
+ // configured timezone (Asia/Shanghai).
+ RangePartitionInfo partitionInfo = (RangePartitionInfo)
table.getPartitionInfo();
+ for (Map.Entry<Long, PartitionItem> entry :
partitionInfo.getIdToItem(false).entrySet()) {
+ RangePartitionItem item = (RangePartitionItem)
entry.getValue();
+ String partitionName =
table.getPartition(entry.getKey()).getName();
+ Assert.assertTrue("Partition name should start with 'p': " +
partitionName,
+ partitionName.startsWith("p"));
+ // Month partition names: p + yyyyMM → length 7
+ Assert.assertEquals("Month partition name length: " +
partitionName,
+ 7, partitionName.length());
+
+ // Verify range validity
+ Range<PartitionKey> range = item.getItems();
+ Assert.assertTrue("lower must be < upper",
+ range.lowerEndpoint().compareTo(range.upperEndpoint())
< 0);
+
+ // Partition boundaries must be UTC timestamps with +00:00
suffix.
+ List<LiteralExpr> lowerKeys = range.lowerEndpoint().getKeys();
+ Assert.assertEquals(1, lowerKeys.size());
+ String lowerStr = lowerKeys.get(0).getStringValue();
+ Assert.assertTrue("Lower key must be UTC with +00:00 suffix: "
+ lowerStr,
+ lowerStr.contains("+00:00"));
+
+ List<LiteralExpr> upperKeys = range.upperEndpoint().getKeys();
+ Assert.assertEquals(1, upperKeys.size());
+ String upperStr = upperKeys.get(0).getStringValue();
+ Assert.assertTrue("Upper key must be UTC with +00:00 suffix: "
+ upperStr,
+ upperStr.contains("+00:00"));
+
+ // Partition boundaries must be at UTC midnight (hour=00)
+ // regardless of time_zone. Day should be 01 (first of month).
+ String lowerDay = lowerStr.substring(8, 10);
+ Assert.assertEquals("Lower bound must be day 01 for month
unit: " + lowerStr,
+ "01", lowerDay);
+ String lowerHour = lowerStr.substring(11, 13);
+ Assert.assertEquals("Lower bound must be UTC midnight (00): "
+ lowerStr,
+ "00", lowerHour);
+ String upperHour = upperStr.substring(11, 13);
+ Assert.assertEquals("Upper bound must be UTC midnight (00): "
+ upperStr,
+ "00", upperHour);
+ }
+
+ // Verify the current partition (idx=0) name matches the UTC
+ // calendar month, since both names and values are UTC-based.
+ // Accept either the current or previous month in case the wall
+ // clock rolled over between scheduler execution and this
assertion.
+ ZonedDateTime utcNow = ZonedDateTime.now(ZoneOffset.UTC);
+ String expectedName = "p" +
DateTimeFormatter.ofPattern("yyyyMM").format(utcNow);
+ String prevExpectedName = "p" +
DateTimeFormatter.ofPattern("yyyyMM")
+ .format(utcNow.minusMonths(1));
+ Assert.assertTrue("Should find month partition '" + expectedName +
"' or '" + prevExpectedName
+ + "' named per UTC",
+ table.getPartitionNames().contains(expectedName)
+ || table.getPartitionNames().contains(prevExpectedName));
+ } finally {
+ connectContext.getSessionVariable().setTimeZone(originalTimeZone);
+ }
+ }
+
+ @Test
+ public void testTimeStampTzDynamicPartitionYearUnit() throws Exception {
+ // Verify TIMESTAMPTZ dynamic partition with YEAR unit:
+ // boundaries are at UTC midnight (month=01, day=01 00:00:00+00:00),
+ // names use the configured timezone's calendar year.
+ String originalTimeZone =
connectContext.getSessionVariable().getTimeZone();
+ try {
+ connectContext.getSessionVariable().setTimeZone("America/Chicago");
+
+ String createOlapTblStmt = "CREATE TABLE
test.`timestamptz_dynamic_year` (\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"true\",\n"
+ + "\"dynamic_partition.time_unit\" = \"year\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" = \"Asia/Shanghai\"\n"
+ + ");";
+ createTable(createOlapTblStmt);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("timestamptz_dynamic_year");
+ Assert.assertTrue(table.dynamicPartitionExists());
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(),
table.getId());
+
+ int partitionCount = table.getPartitionNames().size();
+ Assert.assertEquals(7, partitionCount);
+
+ // Verify partition boundaries are UTC midnight and names use
+ // configured timezone (Asia/Shanghai).
+ RangePartitionInfo partitionInfo = (RangePartitionInfo)
table.getPartitionInfo();
+ for (Map.Entry<Long, PartitionItem> entry :
partitionInfo.getIdToItem(false).entrySet()) {
+ RangePartitionItem item = (RangePartitionItem)
entry.getValue();
+ String partitionName =
table.getPartition(entry.getKey()).getName();
+ Assert.assertTrue("Partition name should start with 'p': " +
partitionName,
+ partitionName.startsWith("p"));
+ // Year partition names: p + yyyy → length 5
+ Assert.assertEquals("Year partition name length: " +
partitionName,
+ 5, partitionName.length());
+
+ // Verify range validity
+ Range<PartitionKey> range = item.getItems();
+ Assert.assertTrue("lower must be < upper",
+ range.lowerEndpoint().compareTo(range.upperEndpoint())
< 0);
+
+ // Partition boundaries must be UTC timestamps with +00:00
suffix.
+ List<LiteralExpr> lowerKeys = range.lowerEndpoint().getKeys();
+ Assert.assertEquals(1, lowerKeys.size());
+ String lowerStr = lowerKeys.get(0).getStringValue();
+ Assert.assertTrue("Lower key must be UTC with +00:00 suffix: "
+ lowerStr,
+ lowerStr.contains("+00:00"));
+
+ List<LiteralExpr> upperKeys = range.upperEndpoint().getKeys();
+ Assert.assertEquals(1, upperKeys.size());
+ String upperStr = upperKeys.get(0).getStringValue();
+ Assert.assertTrue("Upper key must be UTC with +00:00 suffix: "
+ upperStr,
+ upperStr.contains("+00:00"));
+
+ // Partition boundaries must be at UTC midnight (hour=00)
+ // regardless of time_zone. Month should be 01, day should be
01.
+ String lowerMonth = lowerStr.substring(5, 7);
+ Assert.assertEquals("Lower bound must be month 01 for year
unit: " + lowerStr,
+ "01", lowerMonth);
+ String lowerDay = lowerStr.substring(8, 10);
+ Assert.assertEquals("Lower bound must be day 01 for year unit:
" + lowerStr,
+ "01", lowerDay);
+ String lowerHour = lowerStr.substring(11, 13);
+ Assert.assertEquals("Lower bound must be UTC midnight (00): "
+ lowerStr,
+ "00", lowerHour);
+ String upperHour = upperStr.substring(11, 13);
+ Assert.assertEquals("Upper bound must be UTC midnight (00): "
+ upperStr,
+ "00", upperHour);
+ }
+
+ // Verify the current partition (idx=0) name matches the UTC
+ // calendar year, since both names and values are UTC-based.
+ // Accept either the current or previous year in case the wall
+ // clock rolled over between scheduler execution and this
assertion.
+ ZonedDateTime utcNow = ZonedDateTime.now(ZoneOffset.UTC);
+ String expectedName = "p" +
DateTimeFormatter.ofPattern("yyyy").format(utcNow);
+ String prevExpectedName = "p" + DateTimeFormatter.ofPattern("yyyy")
+ .format(utcNow.minusYears(1));
+ Assert.assertTrue("Should find year partition '" + expectedName +
"' or '" + prevExpectedName
+ + "' named per UTC",
+ table.getPartitionNames().contains(expectedName)
+ || table.getPartitionNames().contains(prevExpectedName));
+ } finally {
+ connectContext.getSessionVariable().setTimeZone(originalTimeZone);
+ }
+ }
+
+ @Test
+ public void testTimeStampTzReservedHistoryPeriodsUtcAligned() throws
Exception {
+ // Verify that reserved_history_periods uses UTC-midnight boundaries
+ // (via borderTimeZone from getDropPartitionOpForDynamic) consistently
+ // with the main drop cutoff and add-partition boundaries.
+ // Without the fix, getClosedRange() hardcoded the configured timezone
+ // (e.g. Asia/Shanghai) for convertToUtcTimestamp(), shifting reserved
+ // ranges by the UTC offset and causing mismatches with UTC-aligned
+ // partition boundaries.
+ String originalTimeZone =
connectContext.getSessionVariable().getTimeZone();
+ try {
+ connectContext.getSessionVariable().setTimeZone("America/Chicago");
+
+ String createSql = "CREATE TABLE test.`tstz_reserved_hist` (\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"false\",\n"
+ + "\"dynamic_partition.time_unit\" = \"day\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" = \"Asia/Shanghai\",\n"
+ + "\"dynamic_partition.reserved_history_periods\" =
\"[2019-06-01,2020-08-01]\"\n"
+ + ");";
+ createTable(createSql);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("tstz_reserved_hist");
+
+ // Add partitions manually — must temporarily disable dynamic
partition.
+ alterTable("ALTER TABLE test.tstz_reserved_hist SET "
+ + "('dynamic_partition.enable' = 'false')");
+ // Add a partition that falls within the reserved period with
+ // UTC-midnight boundaries: [2020-01-01 00:00 UTC, 2020-02-01
00:00 UTC).
+ // This should be protected by the reserved period.
+ alterTable("ALTER TABLE test.tstz_reserved_hist ADD PARTITION
p_202001 VALUES "
+ + "[('2020-01-01 00:00:00+00:00'), ('2020-02-01
00:00:00+00:00'))");
+ // Add a partition entirely before the reserved period.
+ // This should be dropped by the start=-3 cutoff.
+ alterTable("ALTER TABLE test.tstz_reserved_hist ADD PARTITION
p_old VALUES "
+ + "[('2018-01-01 00:00:00+00:00'), ('2018-02-01
00:00:00+00:00'))");
+
+ Assert.assertTrue("p_202001 should exist before scheduling",
+ table.getPartitionNames().contains("p_202001"));
+ Assert.assertTrue("p_old should exist before scheduling",
+ table.getPartitionNames().contains("p_old"));
+
+ // Re-enable dynamic partition and run the scheduler.
+ alterTable("ALTER TABLE test.tstz_reserved_hist SET "
+ + "('dynamic_partition.enable' = 'true')");
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(),
table.getId());
+
+ // p_202001 falls within reserved period [2019-06-01, 2020-08-01]
in
+ // UTC-midnight interpretation → must be kept.
+ Assert.assertTrue("p_202001 should be kept — within UTC-aligned
reserved history period",
+ table.getPartitionNames().contains("p_202001"));
+ // p_old is before the start=-3 cutoff and outside the reserved
+ // period → must be dropped.
+ Assert.assertFalse("p_old should be dropped — outside reserved
period and before start",
+ table.getPartitionNames().contains("p_old"));
+ } finally {
+ connectContext.getSessionVariable().setTimeZone(originalTimeZone);
+ }
+ }
+
+ @Test
+ public void testTimeStampTzHotPartitionCooldownUtcAligned() throws
Exception {
+ // Verify that hot-partition cooldown times are UTC-midnight aligned
+ // and not shifted by the configured timezone (America/Chicago, UTC-5).
+ // Without the fix, setStorageMediumProperty() used nowTz (configured
+ // timezone) which could place the cooldown boundary on a different
+ // calendar day than the partition's UTC range.
+ String originalTimeZone =
connectContext.getSessionVariable().getTimeZone();
+ try {
+ connectContext.getSessionVariable().setTimeZone("Asia/Shanghai");
+ changeBeDisk(TStorageMedium.SSD);
+
+ String createSql = "CREATE TABLE test.`tstz_hot_part_cooldown` (\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"true\",\n"
+ + "\"dynamic_partition.time_unit\" = \"day\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" =
\"America/Chicago\",\n"
+ + "\"dynamic_partition.hot_partition_num\" = \"1\"\n"
+ + ");";
+ createTable(createSql);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("tstz_hot_part_cooldown");
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(),
table.getId());
+
+ RangePartitionInfo partitionInfo = (RangePartitionInfo)
table.getPartitionInfo();
+ // Sort partitions by lower bound.
+ List<Map.Entry<Long, PartitionItem>> sortedEntries =
Lists.newArrayList(
+ partitionInfo.getIdToItem(false).entrySet());
+ sortedEntries.sort((a, b) -> {
+ RangePartitionItem ai = (RangePartitionItem) a.getValue();
+ RangePartitionItem bi = (RangePartitionItem) b.getValue();
+ return
ai.getItems().lowerEndpoint().compareTo(bi.getItems().lowerEndpoint());
+ });
+
+ Assert.assertEquals(7, sortedEntries.size());
+
+ // Partitions idx=-3,-2,-1 are before the hot range → HDD.
+ // Partitions idx=0..3 (4 partitions) are within
hot_partition_num=1:
+ // - idx=0: hot (offset + hotPartitionNum = 0+1 = 1 > 0) → SSD
+ // - idx=1,2,3: hot (offset + hotPartitionNum > 0) → SSD
+ // Because hot_partition_num=1 means only the current partition is
hot,
+ // but the logic counts from idx + hotPartitionNum > 0.
+ // Actually: the condition is `offset + hotPartitionNum <= 0` →
HDD.
+ // So idx=-3,-2,-1: -3+1=-2≤0 HDD, -2+1=-1≤0 HDD, -1+1=0≤0 HDD.
+ // idx=0,1,2,3: 0+1=1>0 SSD, etc.
+ // idx=-3,-2,-1 are HDD, idx=0,1,2,3 are SSD.
+
+ for (int i = 0; i < sortedEntries.size(); i++) {
+ Map.Entry<Long, PartitionItem> entry = sortedEntries.get(i);
+ RangePartitionItem item = (RangePartitionItem)
entry.getValue();
+ DataProperty dp =
partitionInfo.getDataProperty(entry.getKey());
+
+ if (i < 3) {
+ // Historical partitions: HDD
+ Assert.assertEquals("Historical partition should be HDD:
idx=" + (i - 3),
+ TStorageMedium.HDD, dp.getStorageMedium());
+ } else {
+ // Hot partitions: SSD
+ Assert.assertEquals("Hot partition should be SSD: idx=" +
(i - 3),
+ TStorageMedium.SSD, dp.getStorageMedium());
+
+ // Every hot partition must have a finite cooldown equal to
+ // that partition's upper endpoint (offset +
hotPartitionNum).
+ // Assert it is NOT the MAX fallback, which would indicate
+ // the TIMESTAMPTZ lifecycle string was rejected during
parse.
+ Assert.assertNotEquals("Hot partition must have a finite
cooldown: idx=" + (i - 3),
+ DataProperty.MAX_COOLDOWN_TIME_MS,
dp.getCooldownTimeMs());
+
+ ZonedDateTime cooldownUtc = ZonedDateTime.ofInstant(
+
java.time.Instant.ofEpochMilli(dp.getCooldownTimeMs()),
+ ZoneOffset.UTC);
+
+ // Parse the partition's upper bound as a UTC instant.
+ String upperStr =
item.getItems().upperEndpoint().getKeys().get(0).getStringValue();
+ ZonedDateTime upperUtc = ZonedDateTime.parse(upperStr,
+ DateTimeFormatter.ofPattern("yyyy-MM-dd
HH:mm:ssXXX"));
+
+ Assert.assertEquals("Cooldown must equal the partition
upper bound ("
+ + upperUtc + "): idx=" + (i - 3),
+ upperUtc.toInstant(), cooldownUtc.toInstant());
+ }
+
+ // Verify partition boundaries are UTC midnight.
+ List<LiteralExpr> lowerKeys =
item.getItems().lowerEndpoint().getKeys();
+ String lowerStr = lowerKeys.get(0).getStringValue();
+ Assert.assertTrue("Lower key must be UTC: " + lowerStr,
+ lowerStr.contains("+00:00"));
+ Assert.assertEquals("Lower bound must be at UTC midnight: " +
lowerStr,
+ "00", lowerStr.substring(11, 13));
+ }
+ } finally {
+ connectContext.getSessionVariable().setTimeZone(originalTimeZone);
+ }
+ }
+
+ @Test
+ public void testTimeStampTzHotPartitionCooldownDstMonth() throws Exception
{
+ // Regression: When the session timezone has DST (e.g. America/Chicago,
+ // UTC-5 in summer, UTC-6 in winter),
PropertyAnalyzer.analyzeDataProperty()
+ // parses cooldown values as DATETIME via
DateLiteralUtils.createDateLiteral()
+ // which applies the current Instant's offset during the initial shift
+ // while using the target date's offset in unixTimestamp(). With the
old
+ // convertToUtcTimestamp path (which appended "+00:00"), a cooldown
+ // boundary in a different DST period than NOW would shift by one hour.
+ // The fix converts the UTC cooldown to wall-clock time in the session
+ // timezone, so both the initial shift and unixTimestamp() use
consistent
+ // offsets. This test uses MONTH unit with America/Chicago (DST) for
+ // both session and partition timezones so the cooldown boundary is
+ // likely to fall in a different DST period than the current date.
+ String originalTimeZone =
connectContext.getSessionVariable().getTimeZone();
+ try {
+ connectContext.getSessionVariable().setTimeZone("America/Chicago");
+ changeBeDisk(TStorageMedium.SSD);
+
+ String createSql = "CREATE TABLE test.`tstz_cooldown_dst_month`
(\n"
+ + " `k1` TIMESTAMPTZ NULL COMMENT \"\",\n"
+ + " `k2` int NULL COMMENT \"\"\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(`k1`, `k2`)\n"
+ + "PARTITION BY RANGE(`k1`)\n"
+ + "()\n"
+ + "DISTRIBUTED BY HASH(`k2`) BUCKETS 3\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\",\n"
+ + "\"dynamic_partition.enable\" = \"true\",\n"
+ + "\"dynamic_partition.start\" = \"-3\",\n"
+ + "\"dynamic_partition.end\" = \"3\",\n"
+ + "\"dynamic_partition.create_history_partition\" =
\"true\",\n"
+ + "\"dynamic_partition.time_unit\" = \"month\",\n"
+ + "\"dynamic_partition.prefix\" = \"p\",\n"
+ + "\"dynamic_partition.buckets\" = \"1\",\n"
+ + "\"dynamic_partition.time_zone\" =
\"America/Chicago\",\n"
+ + "\"dynamic_partition.hot_partition_num\" = \"6\"\n"
+ + ");";
+ createTable(createSql);
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
+ OlapTable table = (OlapTable)
db.getTableOrAnalysisException("tstz_cooldown_dst_month");
+
+ Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .executeDynamicPartitionFirstTime(db.getId(),
table.getId());
+
+ RangePartitionInfo partitionInfo = (RangePartitionInfo)
table.getPartitionInfo();
+ List<Map.Entry<Long, PartitionItem>> sortedEntries =
Lists.newArrayList(
+ partitionInfo.getIdToItem(false).entrySet());
+ sortedEntries.sort((a, b) -> {
+ RangePartitionItem ai = (RangePartitionItem) a.getValue();
+ RangePartitionItem bi = (RangePartitionItem) b.getValue();
+ return
ai.getItems().lowerEndpoint().compareTo(bi.getItems().lowerEndpoint());
+ });
+
+ Assert.assertEquals(7, sortedEntries.size());
+
+ // idx=-3,-2,-1: offset+6=3,2,1 >0 → SSD (hot)
+ // idx=0..3: offset+6=6,7,8,9 >0 → SSD (hot)
+ // All partitions should be SSD since hot_partition_num covers all.
+ DynamicPartitionProperty prop =
table.getTableProperty().getDynamicPartitionProperty();
+ for (int i = 0; i < sortedEntries.size(); i++) {
+ int idx = i - 3; // idx=-3,-2,-1,0,1,2,3
+ Map.Entry<Long, PartitionItem> entry = sortedEntries.get(i);
+ RangePartitionItem item = (RangePartitionItem)
entry.getValue();
+ DataProperty dp =
partitionInfo.getDataProperty(entry.getKey());
+
+ Assert.assertEquals("All partitions should be SSD with
hot_partition_num=6",
+ TStorageMedium.SSD, dp.getStorageMedium());
+ Assert.assertNotEquals("Hot partition must have a finite
cooldown",
+ DataProperty.MAX_COOLDOWN_TIME_MS,
dp.getCooldownTimeMs());
+
+ // Cooldown is the lower bound of the (idx +
hotPartitionNum)-th
+ // partition. Verify it matches what the scheduler computes.
+ ZonedDateTime cooldownUtc = ZonedDateTime.ofInstant(
+ java.time.Instant.ofEpochMilli(dp.getCooldownTimeMs()),
+ ZoneOffset.UTC);
+ ZonedDateTime utcNow = ZonedDateTime.now(ZoneOffset.UTC);
+ String partFormat = DynamicPartitionUtil.getPartitionFormat(
Review Comment:
This expected cooldown basis is sampled after the scheduler has already
generated the partitions. `executeDynamicPartitionFirstTime()` uses the `now`
captured inside `getAddPartitionOp()` when `setStorageMediumProperty()` formats
each cooldown, but this assertion then calls
`ZonedDateTime.now(ZoneOffset.UTC)` again and derives `idx + 6` month borders
from that later instant. If the scheduler runs just before a UTC month boundary
and this assertion runs just after it, the actual cooldowns are based on the
previous month while `expectedCooldownBorder` is based on the new month, so the
test fails even though the production path used one consistent clock. Please
derive the expected border from the generated partition ranges/current
partition, or otherwise share the same captured clock with the scheduler path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
