nsivabalan commented on code in PR #7337:
URL: https://github.com/apache/hudi/pull/7337#discussion_r1035479716


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -711,15 +713,16 @@ private void initializeFileGroups(HoodieTableMetaClient 
dataMetaClient, Metadata
     LOG.info(String.format("Creating %d file groups for partition %s with base 
fileId %s at instant time %s",
         fileGroupCount, metadataPartition.getPartitionPath(), 
metadataPartition.getFileIdPrefix(), instantTime));
     for (int i = 0; i < fileGroupCount; ++i) {
-      final String fileGroupFileId = String.format("%s%04d", 
metadataPartition.getFileIdPrefix(), i);
+      // Suffix of -0 is used as the file-index (see HoodieCreateHandle) as we 
want a single file per fileGroup

Review Comment:
   Note to reviewer: Please note the change in fileId format. 



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -162,8 +170,10 @@ protected void commit(String instantTime, 
Map<MetadataPartitionType, HoodieData<
         // clean plan is the same, so we don't need to delete the requested 
and inflight instant
         // files in the active timeline.
       }
-      
-      List<WriteStatus> statuses = 
writeClient.upsertPreppedRecords(preppedRecordRDD, instantTime).collect();
+
+      List<WriteStatus> statuses = partitioner.isPresent()

Review Comment:
   we are using partitioner to deduce whether its first commit or subsequent 
one. 
   wondering if we can add a new commitMethod altogether only for initial 
commit and keep it light weight (we don't need to trigger compaction, cleaning 
etc) and directly do bulk_insert for eg.
   



##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -258,6 +268,30 @@ public static boolean metadataPartitionExists(String 
basePath, HoodieEngineConte
     }
   }
 
+  /**
+   * Tag each record with the location in the given partition.
+   *
+   * The record is tagged with respective file slice's location based on its 
record key.
+   */
+  public static HoodieData<HoodieRecord> 
tagRecordsWithLocation(HoodieTableMetaClient metadataMetaClient, 
HoodieData<HoodieRecord> records, String partitionName) {

Review Comment:
   Note to reviewer: tagging the records upfront here w/ actual location



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -189,24 +187,28 @@ private void enablePartitions() {
     final HoodieMetadataConfig metadataConfig = 
dataWriteConfig.getMetadataConfig();
     boolean isBootstrapCompleted;
     Option<HoodieTableMetaClient> metaClient = Option.empty();
-    try {
-      isBootstrapCompleted = dataMetaClient.getFs().exists(new 
Path(metadataWriteConfig.getBasePath(), HoodieTableMetaClient.METAFOLDER_NAME));
-      if (isBootstrapCompleted) {
-        metaClient = 
Option.of(HoodieTableMetaClient.builder().setConf(hadoopConf.get())
-            .setBasePath(metadataWriteConfig.getBasePath()).build());
-      }
-    } catch (IOException e) {
-      throw new HoodieException("Failed to enable metadata partitions!", e);
+    // TODO: check for metadata/hoodie.properties instead of just metadata dir.
+    isBootstrapCompleted = 
dataMetaClient.getTableConfig().isMetadataTableEnabled();
+    if (isBootstrapCompleted) {
+      metaClient = 
Option.of(HoodieTableMetaClient.builder().setConf(hadoopConf.get())
+          .setBasePath(metadataWriteConfig.getBasePath()).build());
     }
 
     Option<HoodieTableFileSystemView> fsView = Option.ofNullable(
         metaClient.isPresent() ? 
HoodieTableMetadataUtil.getFileSystemView(metaClient.get()) : null);
     enablePartition(MetadataPartitionType.FILES, metadataConfig, metaClient, 
fsView, isBootstrapCompleted);
-    if (metadataConfig.isBloomFilterIndexEnabled()) {
-      enablePartition(MetadataPartitionType.BLOOM_FILTERS, metadataConfig, 
metaClient, fsView, isBootstrapCompleted);
-    }
-    if (metadataConfig.isColumnStatsIndexEnabled()) {
-      enablePartition(MetadataPartitionType.COLUMN_STATS, metadataConfig, 
metaClient, fsView, isBootstrapCompleted);
+    // to initialize new metadata partitions, since we use bulk_insert 
prepped, we can only initialize one new metadata partition at a time.

Review Comment:
   Note to reviewer: Need to consult w/ prashanth as to why we have this new 
requirement (i.e. initializing only one partition at a time w/ MDT)



##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -110,6 +113,13 @@ public class HoodieTableMetadataUtil {
   public static final String PARTITION_NAME_COLUMN_STATS = "column_stats";
   public static final String PARTITION_NAME_BLOOM_FILTERS = "bloom_filters";
 
+  // Suffix to use for bootstrapping additional indexes. Should be less than 
other suffixes.
+  private static final String INDEX_BOOTSTRAP_TIMESTAMP_SUFFIX = "001";
+  // Suffix to use for compaction
+  private static final String COMPACTION_TIMESTAMP_SUFFIX = "002";

Review Comment:
   need to consult w/ Prashant here. as per master, compaction has suffix of 
"001", but w/ this patch we are switching to "002". 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -967,35 +988,39 @@ protected HoodieData<HoodieRecord> 
prepRecords(Map<MetadataPartitionType,
       HoodieData<HoodieRecord>> partitionRecordsMap) {
     // The result set
     HoodieData<HoodieRecord> allPartitionRecords = 
engineContext.emptyHoodieData();
-
-    HoodieTableFileSystemView fsView = 
HoodieTableMetadataUtil.getFileSystemView(metadataMetaClient);

Review Comment:
   Note to reviewer: since the records are tagged upfront, we don't need these. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to