[ https://issues.apache.org/jira/browse/CARBONDATA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiayu Shen updated CARBONDATA-4347: ----------------------------------- Description: We printed a stack when delete multiple segments, got !https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2022/6/6/s00494122/40313f6afbd24bdcbb2239bdaf054272/image.png! When delete multi segments,{_}deletePhysicalPartition{_} will be called each segment and _deleteEmptyPartitionFolders_ will be called with each carbonindex file. But location.getParent (is the partition directory) may be the same between two segments. So there is repetitive action. {code:java} for (Map.Entry<String, List<String>> entry : locationMap.entrySet()) { if (partitionSpecs != null) { Path location = new Path(entry.getKey()); boolean exists = pathExistsInPartitionSpec(partitionSpecs, location); if (!exists) { FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString())); for (String carbonDataFile : entry.getValue()) { FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(carbonDataFile)); } } CarbonFile path = FileFactory.getCarbonFile(location.getParent().toString()); deleteEmptyPartitionFolders(path); } } {code} Try to collect all partition directories which is relative to deleted segments, delete them after segments deleted. > Improve performance when delete empty partition directory > --------------------------------------------------------- > > Key: CARBONDATA-4347 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4347 > Project: CarbonData > Issue Type: Improvement > Reporter: Jiayu Shen > Priority: Major > Fix For: 2.3.1 > > > We printed a stack when delete multiple segments, got > !https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2022/6/6/s00494122/40313f6afbd24bdcbb2239bdaf054272/image.png! > > When delete multi segments,{_}deletePhysicalPartition{_} will be called each > segment and > _deleteEmptyPartitionFolders_ will be called with each carbonindex file. But > location.getParent (is the partition directory) may be the same between two > segments. So there is repetitive action. > > {code:java} > for (Map.Entry<String, List<String>> entry : locationMap.entrySet()) { > if (partitionSpecs != null) { > Path location = new Path(entry.getKey()); > boolean exists = pathExistsInPartitionSpec(partitionSpecs, location); > if (!exists) { > > FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString())); > for (String carbonDataFile : entry.getValue()) { > > FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(carbonDataFile)); > } > } > CarbonFile path = > FileFactory.getCarbonFile(location.getParent().toString()); > deleteEmptyPartitionFolders(path); > } > } {code} > Try to collect all partition directories which is relative to deleted > segments, delete them after segments deleted. > -- This message was sent by Atlassian Jira (v8.20.10#820010)