On second thought I am not entirely sure whether that bug is the issue. Are
you continuously appending to the file that you have copied to the
directory? Because filestream works correctly when the files are atomically
moved to the monitored directory.
TD
On Mon, Jul 14, 2014 at 9:08 PM,
Oh yes, this was a bug and it has been fixed. Checkout from the master
branch!
https://issues.apache.org/jira/browse/SPARK-2362?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20Streaming%20ORDER%20BY%20created%20DESC%2C%20priority%20ASC
TD
On Mon, Jul
Hi Team,
Is this issue with JavaStreamingContext.textFileStream(hdfsfolderpath)
API also? Please conform. If yes, could you please help me to fix this
issue. I'm using spark 1.0.0 version.
Regards,
Rajesh
On Tue, Jul 15, 2014 at 5:42 AM, Tathagata Das tathagata.das1...@gmail.com
wrote:
Oh
I have a basic spark streaming job that is watching a folder, processing
any new file and updating a column family in cassandra using the new
cassandra-spark-driver.
I think there is a problem with SparkStreamingContext.textFileStream... if
I start my job in local mode with no files in the folder
