Re: [Discussion] Improve the reading/writing performance on the big tablestatus file
Hi David, a) Compressing table status is good. But need to check the decompression overhead and how much overall benefit we can get. b) I suggest we can keep multiple 10MB files (or configurable), then read it distributed way. c) Once read all the table status files better to cache them at driver with multilevel hash map. [first level being status of the segment and second level is segment id] Thanks, Ajantha On Fri, Sep 4, 2020 at 10:19 AM akashrn5 wrote: > Hi David, > > After discussing with you its little bit clear, let me just summarize in > some lines > > *Goals* > 1. reduce the size of status file (which reduces overall size wit some MBs) > 2. make table status file less prone to failures, and fast reading during > read > > *For the above goals with your solutions* > > 1. use the compressor, compress the table status file, so that during read > inmemory read happens and > it will faster > 2. to make less prone to failure, *+1 for solution3* , which can combined > with little bit of solution2 (for new format of table status and trace > folder structure ) and solution3 of delta file, to make the read and write > separate so that the read will be faster and it will help to avoid failures > in case of reliability. > > Suggestion: One more point is to maintain the cache of details after forst > read, instead of reading every time, only once the status-uuid is updated > we > can read again, till then we can read from cache, this will help in faster > read and help in our query. > > I suggest you to create a *jira and prepare a design document*, there we > can > cover many impact areas and *avoid fixing small bugs after implementation.* > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
Re: [Discussion] Segment management enhance
Hi David, a) Recently we tested huge concurrent load and compactions but never faced two loads using same segment id issue (because of table status lock in recordNewLoadMetadata), so I am not sure whether we really need to update to UUID. b) And about other segment interfaces, we have to refactor it. It is long pending. Refactor such that we can support TIME TRAVEL. I have to analyze more on this. If somebody has already done some analysis can use thread to refactor the segment interface discussion. Thanks, Ajantha On Fri, Sep 4, 2020 at 1:11 PM Kunal Kapoor wrote: > Hi David, > Then better we keep a mapping for the segment UUID to virtual segment > number in the table status file as well, > Any API through which the user can get the segment details should return > the virtual segment id instead of the UUID. > > On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang > wrote: > > > Hi Kunal, > > > >1. The user uses SQL API or other interfaces. This UUID is a > transaction > > id, and we already stored the timestamp and other informations in the > > segment metadata. > >This transaction id can be used in the loading/compaction/update > > operation. We can append this id into the log if needed. > >Git commit id also uses UUID, so we can consider to use it. What > > information do you want to get from the folder name? > > > >2. It is easy to fix the show segment command's issue. Maybe we can > sort > > segment by timestamp and UUID to generate the index id. The user can > > continue to use it in other commands. > > > > > > > > - > > Best Regards > > David Cai > > -- > > Sent from: > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > >
Re: [Discussion] Update feature enhancement
Hi David. Thanks for proposing this. *+1 from my side.* I have seen users with 200K segments table stored in cloud. It will be really slow to reload all the segments where update happened for indexes like SI, min-max, MV. So, it is good to write as a new segment and just load new segment indexes. (try to reuse this flow UpdateTableModel.loadAsNewSegment = true) and user can compact the segments to avoid many new segments created by update. and we can also move the compacted segments to table status history I guess to avoid more entries in table status. Thanks, Ajantha On Fri, Sep 4, 2020 at 1:48 PM David CaiQiang wrote: > Hi Akash, > > 3. Update operation contain a insert operation. Update operation will > do the same thing how the insert operation process this issue. > > > > - > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
Re: [Discussion] Update feature enhancement
Hi Akash, 3. Update operation contain a insert operation. Update operation will do the same thing how the insert operation process this issue. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Segment management enhance
Hi Kunal, 1. The user uses SQL API or other interfaces. This UUID is a transaction id, and we already stored the timestamp and other informations in the segment metadata. This transaction id can be used in the loading/compaction/update operation. We can append this id into the log if needed. Git commit id also uses UUID, so we can consider to use it. What information do you want to get from the folder name? 2. It is easy to fix the show segment command's issue. Maybe we can sort segment by timestamp and UUID to generate the index id. The user can continue to use it in other commands. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Update feature enhancement
Hi David, 1. Yeah i already told that it will come in to picture in delete case, as update is (delete + insert). 2. yes, we will be loading the single merge file into cache, which can be little bit better compared to existing one. 3. I didnt get the complete ans actually, when exactly you plan to compact those and how to take care the increasing entries in the table status file thanks -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Segment management enhance
Hi David, I don't think changing the segment ID to UUID is a good idea, it will cause usability issues. 1. Seeing a UUID named directory in the table structure would be weird, and not informative. 2. The show segments command would also have the same problem. Thanks Kunal Kapoor On Fri, Sep 4, 2020 at 8:38 AM David CaiQiang wrote: > [Background] > 1. In some scenes, two loading/compaction jobs maybe write data to the same > segment, it will result in some data confusion and impact some features > which will not work fine again. > 2. Loading/compaction/update/delete operations need to clean stale data > before execution. Cleaning stale data is a high-risk operation, if it has > some exception, it will delete valid data. If the system doesn't clean > stale > data, in some scenes, it will be added into a new merged index file and > can be queried. > 3. Loading/compaction takes a long time and lock will keep a long time also > in some scenes. > > [Motivation & Goal] > We should avoid data confusion and the risk of clean stale data. Maybe we > can use UUID as a segment id to avoid these troubles. Even if we can do > loading/compaction without the segment/compaction lock. > > [Modification] > 1. segment id > Using UUID as segment id instead of the unique numeric value. > > 2. segment layout > a) move segment data folder into the table folder > b) move carbonindexmerge file into Metadata/segments folder, > > tableFolder > UUID1 > |_xxx.carbondata > |_xxx.carobnindex > UUID2 > Metadata > |_segemnts > |_UUID1_timestamp1.segment (segment index summary) > |_UUID1_timestamp1.carbonindexmerge (segment index detail) > |_schema > |_tablestatus > LockFiles > > partitionTableFolder > partkey=value1 > |_xxx.carbondata > |_xxx.carobnindex > partkey=value2 > Metadata > |_segemnts > |_UUID1_timestamp1.segment (segment index summary) > |_partkey=value1 > |_UUID1_timestamp1.carbonindexmerge (segment index detail) > |_partkey=value2 > |_schema > |_tablestatus > LockFiles > > 3. segment management > Extracting segment interface, it can support open/close, read/write, and > segment level index pruning API. > The segment should support multiple data source types: file format(carbon, > parquet, orc...), HBase... > > 4. clean stale data > it will become an optional operation. > > > > - > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
Re: [Discussion] Update feature enhancement
Hi Akash, 1. the update operation still has "deletdelta" files, it keeps the same with previous. horizontal compaction is still needed. 2. loading one carbonindexmerge file will fast, and not impact the query performance. (customer has faced this issue) 3. for insert/loading, it can trigger compaction to avoid small segments. - Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Update feature enhancement
Hi david, Please check below points One advantage what we get here is , when we insert as new segment, it will take the new insert flow without converter step and that will be faster. But here are some points. 1. when you write for new segments for each update, the horizontal compaction in case of update does not make sense, as it wont happen with this idea. With this solution, horizontal compaction makes sense only in delete case. 2. you said we avoid reloading the indexes, but we will avoid reloading the indexes of complete segment(original segment on which update has happened), but we still need to reload the index of newly added segment which has updated right. 3. when you keep on adding multiple segments, we will have more number of segments and if we does not do compaction, that's one problem and the entry and size of metadata(table status) increases so much which is another problem. So how are you going to handle these cases? correct me if i'm wrong in my understanding. Regards, Akash -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
