[
https://issues.apache.org/jira/browse/FALCON-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313342#comment-15313342
]
Sowmya Ramesh edited comment on FALCON-1894 at 6/6/16 5:57 PM:
---------------------------------------------------------------
[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it
doesn't use the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.
Below code in HdfsReplicationRecipeTool takes care of generating the absolute
path
{noformat}
// Construct fully qualified hdfs src path
String srcPaths =
recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
.REPLICATION_SOURCE_DIR.getName());
StringBuilder absoluteSrcPaths = new StringBuilder();
String srcFsPath = recipeProperties.getProperty(
HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
if (StringUtils.isNotEmpty(srcFsPath)) {
srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
}
if (StringUtils.isNotEmpty(srcPaths)) {
String[] paths = srcPaths.split(COMMA_SEPARATOR);
for (String path : paths) {
StringBuilder srcpath = new StringBuilder(srcFsPath);
srcpath.append(path.trim());
srcpath.append(COMMA_SEPARATOR);
absoluteSrcPaths.append(srcpath);
}
}
{noformat}
Fix you have recommended will not work if user wants to replicate multiple
comma separated directories.
Better workaround will be for the suer to provide the fully qualified source
path to replicate instead of modifying the workflow template as below
{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/,
hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}
Full qualified paths should be specified only for drSourceDir and not for
drTargetDir.
With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as
extensions and processing is moved to server side. Logic to construct the fully
qualified path is on the server side and hence will work for both UI and cmd
line.
I am closing this as fixed as it will not be sen after 0.10 release.
Thanks!
was (Author: sowmyaramesh):
[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it
doesn't use the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.
Below code in HdfsReplicationRecipeTool takes care of generating the absolute
path
{noformat}
// Construct fully qualified hdfs src path
String srcPaths =
recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
.REPLICATION_SOURCE_DIR.getName());
StringBuilder absoluteSrcPaths = new StringBuilder();
String srcFsPath = recipeProperties.getProperty(
HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
if (StringUtils.isNotEmpty(srcFsPath)) {
srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
}
if (StringUtils.isNotEmpty(srcPaths)) {
String[] paths = srcPaths.split(COMMA_SEPARATOR);
for (String path : paths) {
StringBuilder srcpath = new StringBuilder(srcFsPath);
srcpath.append(path.trim());
srcpath.append(COMMA_SEPARATOR);
absoluteSrcPaths.append(srcpath);
}
}
{noformat}
Fix you have recommended will not work if user wants to replicate multiple
comma separated directories.
Better workaround will be for the suer to provide the fully qualified source
path to replicate instead of modifying the workflow template as below
{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/,
hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}
With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as
extensions and processing is moved to server side. Logic to construct the fully
qualified path is on the server side and hence will work for both UI and cmd
line.
I am closing this as fixed as it will not be sen after 0.10 release.
Thanks!
> HDFS Data replication cannot be initiated independent of Oozie server location
> ------------------------------------------------------------------------------
>
> Key: FALCON-1894
> URL: https://issues.apache.org/jira/browse/FALCON-1894
> Project: Falcon
> Issue Type: Bug
> Components: general
> Affects Versions: trunk
> Reporter: Alex Bush
> Assignee: Sowmya Ramesh
> Priority: Minor
> Fix For: trunk, 0.10
>
>
> The HDFS mirroring scripts allow replication between two clusters.
> Currently, even though the UI allows the replication in any direction between
> clusters independent of which cluster the Falcon and Oozie servers belong to
> this is not observed and the source cluster is always the cluster with
> Oozie/Falcon server.
> Steps to reproduce:
> 1) Define both clusters in Falcon server on cluster 2
> 2) Set up HDFS mirroring in Falcon server on cluster 2 from Cluster 1 to
> Cluster 2 and set to run on Oozie server of Cluster 2
> Result:
> Falcon will replicate data from Cluster 2 to Cluster 2
> Cause:
> In hdfs-replication-workflow.xml, the source dir should be defined like the
> target dir by including clusterfs:
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L63
> <arg>${drSourceDir}</arg>
> should be
> <arg>${drSourceClusterFS}${drSourceDir}</arg>
>
> like
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L65
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
