[
https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112126#comment-17112126
]
Vinoth Chandar commented on HUDI-914:
-------------------------------------
>Although specifying the namenode IP address of the target cluster can be
>written, this loses HDFS high availability
I think you are referring to the fact that the other configs for HA NameNode
won't be picked up for e.g?
I think having a way to explicitly pick up configuration for target cluster in
delta streamer and data source (IIUC you will just be augmenting the
sparkContext with this additional configurations) is a good addition..
> support different target data clusters
> --------------------------------------
>
> Key: HUDI-914
> URL: https://issues.apache.org/jira/browse/HUDI-914
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: DeltaStreamer
> Reporter: liujinhui
> Assignee: liujinhui
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently hudi-DeltaStreamer does not support writing to different target
> clusters. The specific scenarios are as follows: Generally, Hudi tasks run on
> an independent cluster. If you want to write data to the target data cluster,
> you generally rely on core-site.xml and hdfs-site.xml; sometimes you will
> encounter different targets. The data cluster writes data, but the cluster
> running the hudi task does not have the core-site.xml and hdfs-site.xml of
> the target cluster. Although specifying the namenode IP address of the target
> cluster can be written, this loses HDFS high availability, so I plan to Use
> the contents of the core-site.xml and hdfs-site.xml files of the target
> cluster as configuration items and configure them in the
> dfs-source.properties or kafka-source.properties file of Hudi.
> Is there a better way to solve this problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
