@David: a) yes, SI can use global by default. b) Handling SI original load itself to launch task based on SI segment size (need to figure out how to estimate) is better, else we have to go with one task per node logic (similar to main table local sort). But current logic needs to changed to avoid small files problem. c) Refresh Index for SI is currently only for merging the small files, we have to rename this command I think. Naming doesn't make sense. and ReIndex is for loading the missed SI segments from main table, cannot use it for merge.
@Akash: a) Loading time difference between SI global_sort and local_sort is the same as the Data loading difference of any table global sort and local sort. we already have it. b) yes, after implementing new SI load logic (task launch based on segment size), we can compare current with refresh index time. If not much difference we can remove refresh index support for SI. Thanks, Ajantha On Mon, Nov 9, 2020 at 1:04 PM akashrn5 <[email protected]> wrote: > Hi, > > Its better to remove i feel, as lot of code will be avoided and we can do > it > right the first time we do it. > > but please consider below points. > > 1. may be once we can test the time difference of global sort and exiting > local sort load time, may be per segment basis, so that we can have a > overall time difference we can get in load, basically if we can note down > the tradeoff time, that's better for future reference and in user > perspective also. > > 2. Also can you check the refresh index and reload time diff, because we > need to see if all users fine with dropping and recreating again. > > Regards, > Akash > > > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
