[ https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaofeng SHI updated KYLIN-3626: -------------------------------- Component/s: Storage - Parquet Job Engine > Allow customization for Parquet path > ------------------------------------ > > Key: KYLIN-3626 > URL: https://issues.apache.org/jira/browse/KYLIN-3626 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine, Metadata, Storage - Parquet > Reporter: Shaofeng SHI > Assignee: Chao Long > Priority: Major > Fix For: v2.6.0 > > > Due to the difference of HDFS and object store like S3/blob store, Kylin > needs to allow different path patterns for the files when putting to the > underlying storage. > > For example, to HDFS, this pattern is good: > /kylin/metadata_name/cube_name/segment_name/file1.parquet > > But, the above pattern is not good for S3, since S3 uses the first couple > bytes of the path to do the hashing, so we'd better add salt ahead of the > original pathname: > /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet > or do a reverse order: > /segment_name/cube_name/metadata_name/kylin/file1.parquet > > To get the flexibility, we should make this extensible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)