Hi my actual use case is streaming text files in HDFS directory and send it to Kafka please let me know if is there any existing solution for this. Anyways I have the following code
//lets assume directory contains one file a.txt and it has 100 lines JavaDStream<String> logData = jssc.textFileStream(directory); //how do I make sure jssc.textFileStream() read all 100 lines so I can delete it later Also please let me know how do I send above logData to Kafka using Spark streaming -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/jssc-textFileStream-directory-how-to-ensure-it-read-entire-all-incoming-files-tp26181.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
