Disha Talreja created ATLAS-5084:
------------------------------------
Summary: provide a way to transform hdfs paths during atlas import
Key: ATLAS-5084
URL: https://issues.apache.org/jira/browse/ATLAS-5084
Project: Atlas
Issue Type: Improvement
Components: atlas-core
Reporter: Disha Talreja
Assignee: Disha Talreja
Currently, there is no way to pass : in the transformation options during atlas
import(check CDPD-67112 ) which is mandatory for Iceberg Tables as iceberg
replication provides custom location replication on the Target side.This means
let's say you have a table on source at location _*hdfs://srcns:8020/a*_ and
you can replicate that table on the DR side to location
_*hdfs://destns:8020/b*_
and this path a and be doesn't need to be defaultFs or default warehouse
locations.
For this iceberg replication provides a Key:value kind of mapping.
So if you want your tables at location _*a*_ to be replicated at location *b*
on the Target side and location _*b*_ on the source side to replicate to
location _*c*_ on the target side that is possible.
for this user can just provide below mapping
*hdfs://srcns:8020/a = hdfs://destns:8020/b*
*hdfs://srcns:8020/b = hdfs://destns:8020/c*
This transformation we need to pass it to atlas import also but as : is not
supported in transformation options only way to pass this is to break down the
string and pass individual component but that will lead to some other issues.
Like in above case if we pass
_*replace:a:b, replace:b:c*_ then there will be no path _*b*_ which is wrong
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
