I list feature list about segment as follows before starting to re-factory
segment interface.
[table related]
1. get lock for table
lock for tablestatus
lock for updatedTablestatus
2. get lastModifiedTime of table
[segment related]
1. segment datasource
datasource: file format,other datasource
fileformat: carbon,parquet,orc,csv..
catalog type: segment, external segment
2. data load etl(load/insert/add_external_segment/insert_stage)
write segment for batch loading
add external segment by using external folder path for mixed file
formatted table
append streaming segment for spark structed streaming
insert_stage for flink writer
3. data query
segment properties and schema
segment level index cache and pruning
cache/refresh block/blocklet index cache if needed by segment
read segments to a dataframe/rdd
4. segment management
new segment id for loading/insert/add_external_segment/insert_stage
create global segment identifier
show[history]/delete segment
5. stats
collect dataSize and indexSize of the segment
lastModifiedTime, start/end time, update start/end time
fileFormat
status
6. segment level lock for supporting concurrent operations
7. get tablestatus storage factory
storage solution 1): use file system by default
storage solution 2): use hive metastore or db
[table status related]:
1. record new LoadMetadataDetails
loading/insert/compatcion start/end
add external segment start/end
insert stage
2. update LoadMetadataDetails
compation
update/delete
drop partition
delete segment
3. read LoadMetadataDetails
list all/valid/invalid segment
4. backup and history
[segment file related]
1. write new segment file
generate segment file name
better to use new timestamp to generate new segment file name for each
writing. avoid overwriting segment file with same name.
write semgent file
merge temp segment file
2. read segment file
readIndexFiles
readIndexMergeFiles
getPartitionSpec
3. update segment file
update
merge index
drop partition
[clean files related]
1. clean stale files for the successful segment operation
data deletion should delay a period of time(maybe query timeout
interval), avoid deleting file immediately(beside of drop table/partition,
force clean files)
include data file, index file, segment file, tablestatus file
impact operation: mergeIndex
2. clean stale files for failed segment operation immediately
-----
Best Regards
David Cai
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
