Soroka,
You should be able to use the filestream() method of the
JavaStreamingContext. In case you need something more custom, the code
below is something I developed to provide the max functionality of the
Scala method, but implemented in Java.
//Set these to reflect your app and input format specifications. K,V,F are
templates.
StreamingContext ssc = ...;
Class<K> key_class =...;
Class<V> value_class = ...;
Class<F> format_class = ...;
//Setup class tags for Java-Scala interface
ClassTag<K> key_tag = ClassTag$.MODULE$.apply(key_class);
ClassTag<V> value_tag = ClassTag$.MODULE$.apply(value_class);
ClassTag<F> format_tag = ClassTag$.MODULE$.apply(format_class);
ssc.fileStream(directory_with_files, new AbstractFunction1<Path, Object>() {
@Override
public Boolean apply(Path input) { //file filtering logic here }
}, newFilesOnly_boolean, key_tag, value_tag, format_tag);
Mike
On Fri, Dec 19, 2014 at 2:51 PM, soroka21 [via Apache Spark User List] <
ml-node+s1001560n20791...@n3.nabble.com> wrote:
>
> Hello,
>
> I was successfully using my own customized Hadoop InputFormat class with
> JavaSparkContext.newAPIHadoopFile(...)
>
> Is there any way I can reuse my class in Spark Streaming?
>
>
>
> soroka21
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Using-Customized-Hadoop-InputFormat-class-with-Spark-Streaming-tp20791.html
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bXEwMDFrQGdtYWlsLmNvbXwxfDgxMTQwOTE5Nw==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Using-Customized-Hadoop-InputFormat-class-with-Spark-Streaming-tp20791p20793.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
