Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-17 Thread via GitHub


danny0405 merged PR #11545:
URL: https://github.com/apache/hudi/pull/11545


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-16 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2231330230

   
   ## CI report:
   
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 3da3d90ddafde29dcbd08297763be9b406d8d3f7 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24869)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-16 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2231194071

   
   ## CI report:
   
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 03c739e3fa4cef2862343ddbb00844cf748e79ef Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24859)
 
   * 3da3d90ddafde29dcbd08297763be9b406d8d3f7 Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24869)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-16 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2231157694

   
   ## CI report:
   
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 03c739e3fa4cef2862343ddbb00844cf748e79ef Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24859)
 
   * 3da3d90ddafde29dcbd08297763be9b406d8d3f7 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-15 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1678665333


##
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstant.java:
##
@@ -42,9 +42,13 @@ public class HoodieInstant implements Serializable, 
Comparable {
   private static final Pattern NAME_FORMAT =
   Pattern.compile("^(\\d+(_\\d+)?)(\\.\\w+)(\\.\\D+)?$");
 
+  // the naming format of 0.x version
+  public static final Pattern NAME_FORMAT_0_X =
+  Pattern.compile("^(\\d+)(\\.\\w+)(\\.\\D+)?$");

Review Comment:
   If just for testing purposes, let's move this variable just to the test 
class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-15 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2228851484

   
   ## CI report:
   
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 03c739e3fa4cef2862343ddbb00844cf748e79ef Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24859)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-15 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2228708092

   
   ## CI report:
   
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24853)
 
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 03c739e3fa4cef2862343ddbb00844cf748e79ef Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24859)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-15 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2228609611

   
   ## CI report:
   
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24853)
 
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   * 03c739e3fa4cef2862343ddbb00844cf748e79ef UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-15 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2228590641

   
   ## CI report:
   
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24853)
 
   * b940c86e5aa495fd92c5273c5febe727c2b76f99 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1677241737


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java:
##
@@ -20,18 +20,63 @@
 
 import org.apache.hudi.common.config.ConfigProperty;
 import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
 import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
 import java.util.Collections;
+import java.util.List;
 import java.util.Map;
+import java.util.regex.Matcher;
+
+import static org.apache.hudi.common.table.timeline.HoodieInstant.NAME_FORMAT;
+import static org.apache.hudi.common.table.timeline.HoodieInstant.UNDERSCORE;
+
 
 /**
  * Version 7 is going to be placeholder version for bridge release 0.16.0.
  * Version 8 is the placeholder version to track 1.x.
  */
 public class EightToSevenDowngradeHandler implements DowngradeHandler {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(EightToSevenDowngradeHandler.class);
+
   @Override
   public Map downgrade(HoodieWriteConfig config, 
HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade 
upgradeDowngradeHelper) {
+final HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+UpgradeDowngradeUtils.runCompaction(table, context, config, 
upgradeDowngradeHelper);
+UpgradeDowngradeUtils.syncCompactionRequestedFileToAuxiliaryFolder(table);
+
+HoodieTableMetaClient metaClient = 
HoodieTableMetaClient.builder().setConf(context.getStorageConf().newInstance()).setBasePath(config.getBasePath()).build();
+List instants = 
metaClient.getActiveTimeline().getInstants();
+if (!instants.isEmpty()) {
+  context.map(instants, instant -> {
+Matcher matcher = NAME_FORMAT.matcher(instant.getFileName());
+if (matcher.find()) {
+  try {
+// Rename the metadata file name from the 
${instant_time}_${completion_time}.action[.state] format in version 1.x to the 
${instant_time}.action[.state] format in version 0.x.
+StoragePath fromPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName());
+String regexStr = matcher.group(1);
+StoragePath toPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName().replaceAll(regexStr.substring(regexStr.indexOf(UNDERSCORE)),
 ""));

Review Comment:
   Maybe we can make it simpler, first split the file name by dot `.` we get 
the file name and suffix, then split the file name with `_` and we get the 
instant time, then we concatenate the instant time and the suffix and it's done.
   
   And then there is no need to make changes in `HoodieInstant`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1677241737


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java:
##
@@ -20,18 +20,63 @@
 
 import org.apache.hudi.common.config.ConfigProperty;
 import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
 import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
 import java.util.Collections;
+import java.util.List;
 import java.util.Map;
+import java.util.regex.Matcher;
+
+import static org.apache.hudi.common.table.timeline.HoodieInstant.NAME_FORMAT;
+import static org.apache.hudi.common.table.timeline.HoodieInstant.UNDERSCORE;
+
 
 /**
  * Version 7 is going to be placeholder version for bridge release 0.16.0.
  * Version 8 is the placeholder version to track 1.x.
  */
 public class EightToSevenDowngradeHandler implements DowngradeHandler {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(EightToSevenDowngradeHandler.class);
+
   @Override
   public Map downgrade(HoodieWriteConfig config, 
HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade 
upgradeDowngradeHelper) {
+final HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+UpgradeDowngradeUtils.runCompaction(table, context, config, 
upgradeDowngradeHelper);
+UpgradeDowngradeUtils.syncCompactionRequestedFileToAuxiliaryFolder(table);
+
+HoodieTableMetaClient metaClient = 
HoodieTableMetaClient.builder().setConf(context.getStorageConf().newInstance()).setBasePath(config.getBasePath()).build();
+List instants = 
metaClient.getActiveTimeline().getInstants();
+if (!instants.isEmpty()) {
+  context.map(instants, instant -> {
+Matcher matcher = NAME_FORMAT.matcher(instant.getFileName());
+if (matcher.find()) {
+  try {
+// Rename the metadata file name from the 
${instant_time}_${completion_time}.action[.state] format in version 1.x to the 
${instant_time}.action[.state] format in version 0.x.
+StoragePath fromPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName());
+String regexStr = matcher.group(1);
+StoragePath toPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName().replaceAll(regexStr.substring(regexStr.indexOf(UNDERSCORE)),
 ""));

Review Comment:
   Maybe we can make it simpler, first split the file name by dot `.` we get 
the file name and suffix, then split the file name with `_` and we get the 
instant time, then we concatenate the instant time and the suffix and it's done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2227361369

   
   ## CI report:
   
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24853)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2227356690

   
   ## CI report:
   
   * 185f6caa2b16beb587d83eef6c64d5091bf876ad Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24754)
 
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24853)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2227344156

   @codope 
   Thanks for your suggestion. I've already handled everything. Could you 
please review the code again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2227344059

   @codope 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-14 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2227342005

   
   ## CI report:
   
   * 185f6caa2b16beb587d83eef6c64d5091bf876ad Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24754)
 
   * d78c986f373bd2b74594e4fdc3e0f3e53dab0590 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-09 Thread via GitHub


codope commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1671596452


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java:
##
@@ -20,18 +20,53 @@
 
 import org.apache.hudi.common.config.ConfigProperty;
 import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
 import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
 
+import java.io.IOException;
 import java.util.Collections;
+import java.util.List;
 import java.util.Map;
 
+
 /**
  * Version 7 is going to be placeholder version for bridge release 0.16.0.
  * Version 8 is the placeholder version to track 1.x.
  */
 public class EightToSevenDowngradeHandler implements DowngradeHandler {
   @Override
   public Map downgrade(HoodieWriteConfig config, 
HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade 
upgradeDowngradeHelper) {
+final HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+UpgradeDowngradeUtils.runCompaction(table, context, config, 
upgradeDowngradeHelper);
+UpgradeDowngradeUtils.syncCompactionRequestedFileToAuxiliaryFolder(table);
+
+HoodieTableMetaClient metaClient = 
HoodieTableMetaClient.builder().setConf(context.getStorageConf().newInstance()).setBasePath(config.getBasePath()).build();
+List instants = 
metaClient.getActiveTimeline().getInstants();
+if (!instants.isEmpty()) {
+  context.map(instants, instant -> {
+if (!instant.getFileName().contains("_")) {
+  return false;
+}
+try {
+  // Rename the metadata file name from the 
${instant_time}_${completion_time}.action[.state] format in version 1.x to the 
${instant_time}.action[.state] format in version 0.x.
+  StoragePath fromPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName());
+  StoragePath toPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName().replaceAll("_\\d+", ""));
+  boolean success = metaClient.getStorage().rename(fromPath, toPath);
+  // TODO: We need to rename the action-related part of the metadata 
file name here when we bring separate action name for clustering/compaction in 
1.x as well.

Review Comment:
   Is there a separate ticket tracking this TODO?



##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestUpgradeOrDowngradeProcedure.scala:
##
@@ -142,6 +143,56 @@ class TestUpgradeOrDowngradeProcedure extends 
HoodieSparkProcedureTestBase {
 }
   }
 
+  test("Test downgrade table from version eight to version seven") {
+withTempDir { tmp =>
+  val tableName = generateTableName
+  val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+  // create table
+  spark.sql(
+s"""
+   |create table $tableName (
+   |  id int,
+   |  name string,
+   |  price double,
+   |  ts long
+   |) using hudi
+   | location '$tablePath'
+   | options (
+   |  type = 'mor',
+   |  primaryKey = 'id',
+   |  preCombineField = 'ts'
+   | )
+   """.stripMargin)
+
+  spark.sql("set hoodie.compact.inline=true")
+  spark.sql("set hoodie.compact.inline.max.delta.commits=1")
+  spark.sql("set hoodie.clean.commits.retained = 2")
+  spark.sql("set hoodie.keep.min.commits = 3")
+  spark.sql("set hoodie.keep.min.commits = 4")
+  spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+  spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+  spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+  spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+  spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+
+  var metaClient = createMetaClient(spark, tablePath)
+  // verify hoodie.table.version of the table is EIGHT
+  if 
(metaClient.getTableConfig.getTableVersion.versionCode().equals(HoodieTableVersion.EIGHT.versionCode()))
 {
+// downgrade table from version eight to version seven
+checkAnswer(s"""call downgrade_table(table => '$tableName', to_version 
=> 'SEVEN')""")(Seq(true))
+metaClient = HoodieTableMetaClient.reload(metaClient)
+assertResult(HoodieTableVersion.SEVEN.versionCode) {
+  metaClient.getTableConfig.getTableVersion.versionCode()
+}
+// Verify whether the naming format of instant files is consistent 
with 0.x
+
metaClient.reloadActiveTimeline().getInstants.iterator().asScala.forall(f => 
!f.getFileName.contains("_"))

Review Comment:
   Can we add a pat

Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-09 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2219305418

   @danny0405 @codope 
   Hi, masters, all checks has passed. Could you help to review the code? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-08 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2215982081

   @danny0405 Hi, danny. Could you help to review the code? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-07 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2212415234

   
   ## CI report:
   
   * 185f6caa2b16beb587d83eef6c64d5091bf876ad Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24754)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-07 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2212388014

   
   ## CI report:
   
   * 20775fbef5e4ff78cfa45e8497478b2414a39e96 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24718)
 
   * 185f6caa2b16beb587d83eef6c64d5091bf876ad Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24754)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-07 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2212385705

   
   ## CI report:
   
   * 20775fbef5e4ff78cfa45e8497478b2414a39e96 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24718)
 
   * 185f6caa2b16beb587d83eef6c64d5091bf876ad UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-05 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1667211813


##
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java:
##
@@ -338,18 +339,22 @@ protected void deleteInstantFile(HoodieInstant instant) {
*
* TODO: [HUDI-6885] Depreciate HoodieActiveTimeline#getInstantFileName 
and fix related tests.
*/
-  protected String getInstantFileName(HoodieInstant instant) {
-if (instant.isCompleted() && instant.getCompletionTime() == null) {
-  return getInstantsAsStream().filter(s -> s.equals(instant))
-  .findFirst().orElseThrow(() -> new HoodieIOException("Cannot find 
the instant" + instant))
-  .getFileName();
+  protected String getInstantFileName(HoodieInstant instant, int versionCode) {
+if (versionCode < HoodieTableVersion.EIGHT.versionCode()) {

Review Comment:
   It looks like the logic is changed for test code, maybe we should just 
restrict the logic in the tests either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-05 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1667211548


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##
@@ -140,13 +140,19 @@ public void run(HoodieTableVersion toVersion, String 
instantTime) {
 HoodieTableVersion nextVersion = 
HoodieTableVersion.versionFromCode(fromVersion.versionCode() + 1);
 tableProps.putAll(upgrade(fromVersion, nextVersion, instantTime));
 fromVersion = nextVersion;
+metaClient.getTableConfig().setTableVersion(nextVersion);
+HoodieTableConfig.update(metaClient.getStorage(),

Review Comment:
   Why we need this change?



##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##
@@ -140,13 +140,19 @@ public void run(HoodieTableVersion toVersion, String 
instantTime) {
 HoodieTableVersion nextVersion = 
HoodieTableVersion.versionFromCode(fromVersion.versionCode() + 1);
 tableProps.putAll(upgrade(fromVersion, nextVersion, instantTime));
 fromVersion = nextVersion;
+metaClient.getTableConfig().setTableVersion(nextVersion);
+HoodieTableConfig.update(metaClient.getStorage(),
+metaClient.getMetaPath(), 
metaClient.getTableConfig().getProps());
   }
 } else {
   // downgrade
   while (fromVersion.versionCode() > toVersion.versionCode()) {
 HoodieTableVersion prevVersion = 
HoodieTableVersion.versionFromCode(fromVersion.versionCode() - 1);
 tableProps.putAll(downgrade(fromVersion, prevVersion, instantTime));
 fromVersion = prevVersion;
+metaClient.getTableConfig().setTableVersion(prevVersion);
+HoodieTableConfig.update(metaClient.getStorage(),

Review Comment:
   Why we need this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-04 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2209295183

   
   ## CI report:
   
   * 20775fbef5e4ff78cfa45e8497478b2414a39e96 Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24718)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-04 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2209228176

   
   ## CI report:
   
   * 0ea64cd14f03c950f430f53850110fdf39d755ad Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24716)
 
   * 20775fbef5e4ff78cfa45e8497478b2414a39e96 Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24718)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-04 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2209216649

   
   ## CI report:
   
   * 0ea64cd14f03c950f430f53850110fdf39d755ad Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24716)
 
   * 20775fbef5e4ff78cfa45e8497478b2414a39e96 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-04 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2209123123

   
   ## CI report:
   
   * 0ea64cd14f03c950f430f53850110fdf39d755ad Azure: 
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24716)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-04 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2209108930

   
   ## CI report:
   
   * 0e3cb49fb72bdc14dee9e67fe0aaeb0d271608f2 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24698)
 
   * 0ea64cd14f03c950f430f53850110fdf39d755ad UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1665134216


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java:
##
@@ -32,6 +39,28 @@
 public class EightToSevenDowngradeHandler implements DowngradeHandler {
   @Override
   public Map downgrade(HoodieWriteConfig config, 
HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade 
upgradeDowngradeHelper) {
+HoodieTableMetaClient metaClient = 
HoodieTableMetaClient.builder().setConf(context.getStorageConf().newInstance()).setBasePath(config.getBasePath()).build();
+List instants = 
metaClient.getActiveTimeline().getInstants();
+if (!instants.isEmpty()) {
+  context.map(instants, instant -> {
+if (!instant.getFileName().contains("_")) {
+  return false;
+}
+try {
+  // Rename the metadata file name from the 
${instant_time}_${completion_time}.action[.state] format in version 1.x to the 
${instant_time}.action[.state] format in version 0.x.
+  StoragePath fromPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName());
+  StoragePath toPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName().replaceAll("_\\d+", ""));
+  boolean success = metaClient.getStorage().rename(fromPath, toPath);
+  // TODO: We need to rename the action-related part of the metadata 
file name here when we bring separate action name for clustering/compaction in 
1.x as well.
+  if (!success) {
+throw new HoodieIOException("Error when rename the instant file: " 
+ fromPath + " to: " + toPath);
+  }
+  return true;
+} catch (IOException e) {
+  throw new HoodieException("Can not to complete the downgrade from 
version eight to version seven.", e);

Review Comment:
   One thing needs caution here is after the renaming, the file modification 
time has changed, the modification time is used as completion time in 0.x 
branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1665133407


##
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstant.java:
##
@@ -262,8 +259,12 @@ private String getPendingFileName() {
   }
 
   private String getCompleteFileName(String completionTime) {
-ValidationUtils.checkArgument(!StringUtils.isNullOrEmpty(completionTime), 
"Completion time should not be empty");
-String timestampWithCompletionTime = timestamp + "_" + completionTime;
+String timestampWithCompletionTime;
+if (StringUtils.isNullOrEmpty(completionTime)) {

Review Comment:
   When the completion time could be empty then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


danny0405 commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1665132485


##
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstant.java:
##
@@ -136,10 +136,7 @@ public HoodieInstant(StoragePathInfo pathInfo) {
   state = State.COMPLETED;
 }
   }
-  completionTime = timestamps.length > 1
-  ? timestamps[1]
-  // for backward compatibility with 0.x release.
-  : state == State.COMPLETED ? pathInfo.getModificationTime() + "" : 
null;
+  completionTime = timestamps.length > 1 ? timestamps[1] : null;

Review Comment:
   I think we still need to keep these logic, a downgrade logic in there is 
good but we also need to be compatible for 0.x for some code path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2208074798

   @danny0405 Hi, all checks have passed and  all suggestions have resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2208073498

   > @watermelon12138 would you mind to fix the compile errors: 
https://github.com/apache/hudi/actions/runs/9756232191/job/26926142959?pr=11545
   
   @danny0405 @balaji-varadarajan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2208022864

   
   ## CI report:
   
   * 0e3cb49fb72bdc14dee9e67fe0aaeb0d271608f2 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24698)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2207885219

   
   ## CI report:
   
   * fe7aa032f4463035775029ad486ca73ea2d02ac0 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24668)
 
   * 0e3cb49fb72bdc14dee9e67fe0aaeb0d271608f2 Azure: 
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24698)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2207877464

   
   ## CI report:
   
   * fe7aa032f4463035775029ad486ca73ea2d02ac0 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24668)
 
   * 0e3cb49fb72bdc14dee9e67fe0aaeb0d271608f2 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


danny0405 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2207680792

   @watermelon12138 would you mind to fix the compile errors: 
https://github.com/apache/hudi/actions/runs/9756232191/job/26926142959?pr=11545


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


balaji-varadarajan commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2206918238

   > Hi @balaji-varadarajan can you confirm whether the file renaming is needed 
for 0.x release table?
   
   When downgrading from 1.x to 0.x, we need to handle instant file renaming. 
Downgrading is expected to be a disruptive operation. The downgrade script 
requires more changes w.r.t handling archiving and other but can be don in a 
separate PR. cc @nsivabalan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-03 Thread via GitHub


balaji-varadarajan commented on code in PR #11545:
URL: https://github.com/apache/hudi/pull/11545#discussion_r1664568762


##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java:
##
@@ -32,6 +39,26 @@
 public class EightToSevenDowngradeHandler implements DowngradeHandler {
   @Override
   public Map downgrade(HoodieWriteConfig config, 
HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade 
upgradeDowngradeHelper) {
+HoodieTableMetaClient metaClient = 
HoodieTableMetaClient.builder().setConf(context.getStorageConf().newInstance()).setBasePath(config.getBasePath()).build();
+List instants = 
metaClient.getActiveTimeline().getInstants();
+if (!instants.isEmpty()) {
+  context.map(instants, instant -> {
+if (!instant.getFileName().contains("_")) {
+  return false;
+}
+try {
+  StoragePath fromPath = new StoragePath(metaClient.getMetaPath(), 
instant.getFileName());

Review Comment:
   We need to add a todo here when we bring separate action name for 
clustering/compaction in 1.x as well. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-02 Thread via GitHub


danny0405 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2205110034

   Hi @balaji-varadarajan can you confirm whether the file renaming is needed 
for 0.x release table?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-02 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2204909780

   
   ## CI report:
   
   * fe7aa032f4463035775029ad486ca73ea2d02ac0 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24668)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-02 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2204899621

   
   ## CI report:
   
   * fe7aa032f4463035775029ad486ca73ea2d02ac0 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24668)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-02 Thread via GitHub


watermelon12138 commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2204884386

   @hudi-bot run azure
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-02 Thread via GitHub


hudi-bot commented on PR #11545:
URL: https://github.com/apache/hudi/pull/11545#issuecomment-2203114070

   
   ## CI report:
   
   * fe7aa032f4463035775029ad486ca73ea2d02ac0 Azure: 
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24668)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org