marchpure opened a new pull request #3589: [CARBONDATA-3667] Insert stage recover processing of the partition ta… URL: https://github.com/apache/carbondata/pull/3589 …ble throw exception "the unexpected 0 segment found" Why is this PR needed? Recover processing when executing insert stage a partition table throw exception "the unexpected 0 segment found" The reason is the snapshot content of partition table is wrong. What changes were proposed in this PR? Let's review the recover processing's purpose, when the segment is loaded successfully, but fail to clean the stage files, the stage(data) will be loaded again which leads to repetition of data. Before, we expect to save a snapshot file consists of segment id and stage file list, the stage files failed to clean can be clean again when we found the corresponding segment has been loaded successfully. But the segmentid can't be achieve while loading partition table, in the other words, we can't get the right snapshot for partition table. After discussion, we believe that this problem can also be alleviated by adding retry mechanism when deleting stage files. Thus, there are two changes were proposed which is shown below. 1) Remove recover processing of Insert stage command. 2) Add Retry to delete stage files. Does this PR introduce any user interface change? No Is any new testcase added? No ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
