[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386499#comment-14386499 ] Sean Owen commented on SPARK-6401: -- Since it's technically an API change to streaming I'd look for a nod from [~tdas] before proceeding. > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386491#comment-14386491 ] Thomas F. commented on SPARK-6401: -- How do we proceed for this issue ? As we already have in the DStream the saveAsHadoopFiles with historical OutputFormat and saveAsNewAPIHadoopFiles with NewOutputFormat, do we rename StreamingContext.fileStream() into StreamingContext.newAPIHadoopFileStream(with NewInputFormat) and then add hadoopFileStream(with InputFormat) to be completely aligned with Spark Core for hadoop input/output ? Brgds. > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371748#comment-14371748 ] Sean Owen commented on SPARK-6401: -- It's a fair point that the old API is still in Hadoop. This would make for better API consistency but at the expense of a more crowded API. I had imagined that nobody would be developing InputFormats for the old API at this point though. > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371699#comment-14371699 ] Patrick Wendell commented on SPARK-6401: If this is a matter of just adding a simple wrapper, then why not just do it? Hadoop 2.X still supports the 1.X API's and there are legacy integrations there. It doesn't seem like any more work for us since we already support this in the core Spark API's. > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368241#comment-14368241 ] Sean Owen commented on SPARK-6401: -- Yeah it would be more consistent. I suppose I'd be interested to hear whether others think it's worth continuing to add support for it or not though. It's pretty easy to port or carry a parallel version that uses the new InputFormat, right? I think you can even make an adapter. > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367789#comment-14367789 ] Rémy DUBOIS commented on SPARK-6401: Yes I mean the mapred API. All our input formats are developed in mapred API so it would allow us to avoid rewriting them in mapreduce API. Or at least, it would allow us to do it gradually. Since the batch API can read from a mapred InputFormat, don't you think it would be more consistent to have the same possibility in the streaming API? > Unable to load a old API input format in Spark streaming > > > Key: SPARK-6401 > URL: https://issues.apache.org/jira/browse/SPARK-6401 > Project: Spark > Issue Type: Improvement >Reporter: Rémy DUBOIS >Priority: Minor > > The fileStream method of the JavaStreamingContext class does not allow using > a old API InputFormat. > This feature exists in Spark batch but not in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
