[jira] [Commented] (SPARK-1854) Add a version of StreamingContext.fileStream that take hadoop conf object
[
https://issues.apache.org/jira/browse/SPARK-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258552#comment-14258552
]
Tathagata Das commented on SPARK-1854:
--
What I meant was to use the `hadoopConf` object in creating RDDs from the files
discovered as new files.
But I am closing this JIRA because the recommended way by which this was being
used was to set the hadoop configuration was to set all the configurations in
{{sparkContext.hadoopConfiguration}}. This conf will be automatically used
within file stream.
Add a version of StreamingContext.fileStream that take hadoop conf object
-
Key: SPARK-1854
URL: https://issues.apache.org/jira/browse/SPARK-1854
Project: Spark
Issue Type: New Feature
Components: Streaming
Affects Versions: 1.0.0
Reporter: Tathagata Das
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1854) Add a version of StreamingContext.fileStream that take hadoop conf object
[ https://issues.apache.org/jira/browse/SPARK-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254140#comment-14254140 ] Brennon York commented on SPARK-1854: - [~tdas] was going to tackle this issue. Could you assign this to me and provide a little clarity on what exactly you're looking for? I guess I'm only confused in that do you mean to setup a FileInputDStream from the fileStream() method taking a hadoop conf object and a directory, or somehow setup a file stream with *only* the hadoop conf object? Add a version of StreamingContext.fileStream that take hadoop conf object - Key: SPARK-1854 URL: https://issues.apache.org/jira/browse/SPARK-1854 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.0.0 Reporter: Tathagata Das Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
