Re: [Discussion] Segment management enhance
Hi Ramana, I agree with you. When writing segment file, the system use listFiles to collect all index files. In some case, it will add stale index files into segment file. We will try to fix it at first. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Segment management enhance
Hi David, In the current design of data load operation, isolation, consistency is achieved with following steps 1. DataLoad operation needs to check, if it can concurrently execute along with any other operation like Update/delete( most. operations can be allowed, even multiple parallel data loads). 2. Once operation is allowed, then lock is acquired at tablestatus file to create a segment which it continues to load. 3. During load operation, time stamp(long) is used as transaction id for the complete operation. This transaction id uniquely identifies the operation. 4. When data load is complete the operation is committed to table status by taking lock, read & update and release lock. 5. When data load fails then the operation is not committed and data can be cleaned up during failure flow or in case of abrupt process failures, same is cleaned up later. 6. Temporary data with the transaction id is never read or should be discovered by any reader(Ex: never should be discovered using list files etc and used) . All valid data should be read using only committed transaction references. This method of isolation and committing operation is applicable for all the operations like dataload, insert, update or delete. Segment ID does not have much significance in the above flow, though sequence number is used currently, just for convenience. Segment file locks to ensure the atomic commit is must and cannot be avoided, even if we support complete optimistic concurrency control. So segment ID replacing with UUID will not solve the concurrency or data correctness or cleanup or stale data reading issues and also cannot replace locking. There might be some other problem which you need to deep dive. There might be code which is not following the above actions mentioned by me (like discovering files using file listing and without filtering required transaction id), which might be causing issues you have mentioned. Regards, Ramana On Sat, Sep 5, 2020 at 8:31 AM Ajantha Bhat wrote: > Hi David, > > a) Recently we tested huge concurrent load and compactions but never faced > two loads using same segment id issue (because of table status lock in > recordNewLoadMetadata), so I am not sure whether we really need to update > to UUID. > > b) And about other segment interfaces, we have to refactor it. It is long > pending. Refactor such that we can support TIME TRAVEL. I have to analyze > more on this. If somebody has already done some analysis can use thread to > refactor the segment interface discussion. > > Thanks, > Ajantha > > On Fri, Sep 4, 2020 at 1:11 PM Kunal Kapoor > wrote: > > > Hi David, > > Then better we keep a mapping for the segment UUID to virtual segment > > number in the table status file as well, > > Any API through which the user can get the segment details should return > > the virtual segment id instead of the UUID. > > > > On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang > > wrote: > > > > > Hi Kunal, > > > > > >1. The user uses SQL API or other interfaces. This UUID is a > > transaction > > > id, and we already stored the timestamp and other informations in the > > > segment metadata. > > >This transaction id can be used in the loading/compaction/update > > > operation. We can append this id into the log if needed. > > >Git commit id also uses UUID, so we can consider to use it. What > > > information do you want to get from the folder name? > > > > > >2. It is easy to fix the show segment command's issue. Maybe we can > > sort > > > segment by timestamp and UUID to generate the index id. The user can > > > continue to use it in other commands. > > > > > > > > > > > > - > > > Best Regards > > > David Cai > > > -- > > > Sent from: > > > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > > > > >
Re: [Discussion] Segment management enhance
Hi David, a) Recently we tested huge concurrent load and compactions but never faced two loads using same segment id issue (because of table status lock in recordNewLoadMetadata), so I am not sure whether we really need to update to UUID. b) And about other segment interfaces, we have to refactor it. It is long pending. Refactor such that we can support TIME TRAVEL. I have to analyze more on this. If somebody has already done some analysis can use thread to refactor the segment interface discussion. Thanks, Ajantha On Fri, Sep 4, 2020 at 1:11 PM Kunal Kapoor wrote: > Hi David, > Then better we keep a mapping for the segment UUID to virtual segment > number in the table status file as well, > Any API through which the user can get the segment details should return > the virtual segment id instead of the UUID. > > On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang > wrote: > > > Hi Kunal, > > > >1. The user uses SQL API or other interfaces. This UUID is a > transaction > > id, and we already stored the timestamp and other informations in the > > segment metadata. > >This transaction id can be used in the loading/compaction/update > > operation. We can append this id into the log if needed. > >Git commit id also uses UUID, so we can consider to use it. What > > information do you want to get from the folder name? > > > >2. It is easy to fix the show segment command's issue. Maybe we can > sort > > segment by timestamp and UUID to generate the index id. The user can > > continue to use it in other commands. > > > > > > > > - > > Best Regards > > David Cai > > -- > > Sent from: > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > >
Re: [Discussion] Segment management enhance
Hi Kunal, 1. The user uses SQL API or other interfaces. This UUID is a transaction id, and we already stored the timestamp and other informations in the segment metadata. This transaction id can be used in the loading/compaction/update operation. We can append this id into the log if needed. Git commit id also uses UUID, so we can consider to use it. What information do you want to get from the folder name? 2. It is easy to fix the show segment command's issue. Maybe we can sort segment by timestamp and UUID to generate the index id. The user can continue to use it in other commands. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Segment management enhance
Hi David, I don't think changing the segment ID to UUID is a good idea, it will cause usability issues. 1. Seeing a UUID named directory in the table structure would be weird, and not informative. 2. The show segments command would also have the same problem. Thanks Kunal Kapoor On Fri, Sep 4, 2020 at 8:38 AM David CaiQiang wrote: > [Background] > 1. In some scenes, two loading/compaction jobs maybe write data to the same > segment, it will result in some data confusion and impact some features > which will not work fine again. > 2. Loading/compaction/update/delete operations need to clean stale data > before execution. Cleaning stale data is a high-risk operation, if it has > some exception, it will delete valid data. If the system doesn't clean > stale > data, in some scenes, it will be added into a new merged index file and > can be queried. > 3. Loading/compaction takes a long time and lock will keep a long time also > in some scenes. > > [Motivation & Goal] > We should avoid data confusion and the risk of clean stale data. Maybe we > can use UUID as a segment id to avoid these troubles. Even if we can do > loading/compaction without the segment/compaction lock. > > [Modification] > 1. segment id > Using UUID as segment id instead of the unique numeric value. > > 2. segment layout > a) move segment data folder into the table folder > b) move carbonindexmerge file into Metadata/segments folder, > > tableFolder > UUID1 > |_xxx.carbondata > |_xxx.carobnindex > UUID2 > Metadata > |_segemnts > |_UUID1_timestamp1.segment (segment index summary) > |_UUID1_timestamp1.carbonindexmerge (segment index detail) > |_schema > |_tablestatus > LockFiles > > partitionTableFolder > partkey=value1 > |_xxx.carbondata > |_xxx.carobnindex > partkey=value2 > Metadata > |_segemnts > |_UUID1_timestamp1.segment (segment index summary) > |_partkey=value1 > |_UUID1_timestamp1.carbonindexmerge (segment index detail) > |_partkey=value2 > |_schema > |_tablestatus > LockFiles > > 3. segment management > Extracting segment interface, it can support open/close, read/write, and > segment level index pruning API. > The segment should support multiple data source types: file format(carbon, > parquet, orc...), HBase... > > 4. clean stale data > it will become an optional operation. > > > > - > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
[Discussion] Segment management enhance
[Background] 1. In some scenes, two loading/compaction jobs maybe write data to the same segment, it will result in some data confusion and impact some features which will not work fine again. 2. Loading/compaction/update/delete operations need to clean stale data before execution. Cleaning stale data is a high-risk operation, if it has some exception, it will delete valid data. If the system doesn't clean stale data, in some scenes, it will be added into a new merged index file and can be queried. 3. Loading/compaction takes a long time and lock will keep a long time also in some scenes. [Motivation & Goal] We should avoid data confusion and the risk of clean stale data. Maybe we can use UUID as a segment id to avoid these troubles. Even if we can do loading/compaction without the segment/compaction lock. [Modification] 1. segment id Using UUID as segment id instead of the unique numeric value. 2. segment layout a) move segment data folder into the table folder b) move carbonindexmerge file into Metadata/segments folder, tableFolder UUID1 |_xxx.carbondata |_xxx.carobnindex UUID2 Metadata |_segemnts |_UUID1_timestamp1.segment (segment index summary) |_UUID1_timestamp1.carbonindexmerge (segment index detail) |_schema |_tablestatus LockFiles partitionTableFolder partkey=value1 |_xxx.carbondata |_xxx.carobnindex partkey=value2 Metadata |_segemnts |_UUID1_timestamp1.segment (segment index summary) |_partkey=value1 |_UUID1_timestamp1.carbonindexmerge (segment index detail) |_partkey=value2 |_schema |_tablestatus LockFiles 3. segment management Extracting segment interface, it can support open/close, read/write, and segment level index pruning API. The segment should support multiple data source types: file format(carbon, parquet, orc...), HBase... 4. clean stale data it will become an optional operation. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
