Jonathan Vexler created HUDI-6249: ------------------------------------- Summary: Make maps in HoodieTableFileSystemView concurrent maps Key: HUDI-6249 URL: https://issues.apache.org/jira/browse/HUDI-6249 Project: Apache Hudi Issue Type: Improvement Components: timeline-server Reporter: Jonathan Vexler
{code:java} // mapping from partition paths to file groups contained within them protected Map<String, List<HoodieFileGroup>> partitionToFileGroupsMap; /** * PartitionPath + File-Id to pending compaction instant time. */ protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>> fgIdToPendingCompaction; /** * PartitionPath + File-Id to pending compaction instant time. */ protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>> fgIdToPendingLogCompaction; /** * PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped). */ protected Map<HoodieFileGroupId, BootstrapBaseFileMapping> fgIdToBootstrapBaseFile; /** * Track replace time for replaced file groups. */ protected Map<HoodieFileGroupId, HoodieInstant> fgIdToReplaceInstants; /** * Track file groups in pending clustering. */ protected Map<HoodieFileGroupId, HoodieInstant> fgIdToPendingClustering; {code} The maps should be ConcurrentMaps, because addFilesToView in AbstractTableFileSystem can be called on multiple partitions concurrently. SpillableMapBasedFileSystemView uses ExternalSpillableMap which is not thread safe, so we cannot update HoodieTableFileSystemView because then SpillableMapBasedFileSystemView would not implement it correctly -- This message was sent by Atlassian Jira (v8.20.10#820010)