[
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374080#comment-15374080
]
ASF GitHub Bot commented on STORM-1910:
---------------------------------------
GitHub user roshannaik opened a pull request:
https://github.com/apache/storm/pull/1558
STORM-1910 One topology cannot use hdfs spout to read from two locations
Changing the way the spout is configured. Using member functions as the
primary mode of config... still have to supporting the older mode (via conf
object ) for backward compatibility.
Updated test code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/roshannaik/storm STORM-1910
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1558.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1558
----
commit d3ec6021eee07d0f44ce2a3464622e22e6be0c91
Author: Roshan Naik <[email protected]>
Date: 2016-07-13T00:39:56Z
STORM-1910 - One topology cannot use hdfs spout to read from two locations
----
> One topology can't use hdfs spout to read from two locations
> ------------------------------------------------------------
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hdfs
> Affects Versions: 1.0.1
> Reporter: Raghav Kumar Gautam
> Assignee: Roshan Naik
> Fix For: 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
> LOG.error(Configs.SOURCE_DIR + " setting is required");
> throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such
> errors at this time can only be detected at runtime by looking at the worker
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
