Hi Ravi and David, 1. All the automatic clean data in the case of load/insert/compact/delete will be removed, so cleaning will only happen when the clean files command is called.
2. We will only add the data to trash when we try to clean data which is in IN PROGRESS state. In case of COmpacted/Marked For Delete it will not be moved to the trash, it will be directly deleted. The user will only be able to recover the In Progress segments if the user wants. @Ravi -> Is this okay for trash usage? Only using it for in progress segments. 3. No trash management will be implemented, the data will ONLY BE REMOVED from the trash folder immediately when the clean files command is called. There will be no time to live, the data can be kept in the trash folder untill the user triggers clean files command. Let me know if you have any questions. Vikram Ahuja On Fri, Sep 18, 2020 at 1:43 PM David CaiQiang <david.c...@gmail.com> wrote: > agree with Ravindra, > > 1. stop all automatic clean data in load/insert/compact/update/delete... > > 2. when clean files command clean in-progress or uncertain data, we can > move > them to data trash. > it can prevent delete useful data by mistake, we already find this > issue > in some scenes. > other cases(for example clean mark_for_delete/compacted segment) should > not use the data trash folder, clean data directly. > > 3. no need data trash management, suggest keeping it simple. > The clean file command should support empty trash immediately, it will > be enough. > > > > ----- > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
