n3nash commented on issue #2095: URL: https://github.com/apache/hudi/issues/2095#issuecomment-696259058
@vishalpathak1986 Hudi internally maintains an Index to tag the incoming records with the fileId that it maps to. If inserts are written to log files, we require a way from the index to know which log file a particular record was written to. We have indexes such as BloomIndex which are written to parquet files and hence we are able to figure out if a record is present in a parquet file or not, but we don't have such an index for log files. There is no config to turn off writing inserts as parquet, you just have to use an index implementation that can index log files. Currently, only the HbaseIndex can index log files -> https://github.com/apache/hudi/blob/c8e19e2def0c33415bc3945ffb81f524c484c924/hudi-client/src/main/java/org/apache/hudi/index/hbase/HBaseIndex.java#L483. In the future, the record level index I pointed out earlier will be able to index the log files which will eliminate the need of an external K-V store for this feature. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
