nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675261494
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieFileWriterFactory.java
##########
@@ -45,7 +45,8 @@
TaskContextSupplier taskContextSupplier) throws IOException {
final String extension = FSUtils.getFileExtension(path.getName());
if (PARQUET.getFileExtension().equals(extension)) {
- return newParquetFileWriter(instantTime, path, config, schema,
hoodieTable, taskContextSupplier);
+ return newParquetFileWriter(instantTime, path, config, schema,
hoodieTable, taskContextSupplier, config.populateMetaFields(),
Review comment:
As I mentioned in notes, last argument decides whether to enable bloom
filter or not. In this patch, I am haven't fixed all indexes. But it is
possible to fix other index to work for virtual keys. So, added this additional
argument whether to enable bloom filter or not. Even for virtual keyed table,
if one wants to use regular bloom index, they should be able to.
##########
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkCopyOnWriteTable.java
##########
@@ -210,12 +214,20 @@ public void rollbackBootstrap(HoodieEngineContext
context, String instantTime) {
protected HoodieMergeHandle getUpdateHandle(String instantTime, String
partitionPath, String fileId,
Map<String, HoodieRecord<T>> keyToNewRecords, HoodieBaseFile
dataFileToBeMerged) {
+ Option<BaseKeyGenerator> keyGeneratorOpt = Option.empty();
+ if (!config.populateMetaFields()) {
+ try {
+ keyGeneratorOpt = Option.of((BaseKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(new
TypedProperties(config.getProps())));
+ } catch (IOException e) {
+ throw new HoodieIOException("Only BaseKyGenerators are supported when
meta columns are disabled ", e);
Review comment:
not sure what exactly we can say here. We added this new method, where
you can fetch partition path from internal row for row writer path. So, key
generators extending from BaseKeyGenerators only can be used when meta fields
are disabled. Do you have any good suggestion here of what msg to convey.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
