Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

Nice, thank you I'll test this property to see if the error stop; 2016-11-17 14:48 GMT-02:00 Arijit : > Hi Dirceu, > > > For the append issue we are setting "hdfs.append.support" (from Spark code > which reads HDFS configuration) to "true" in hdfs-site.xml and that seemed > to

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

Hi Dirceu, For the append issue we are setting "hdfs.append.support" (from Spark code which reads HDFS configuration) to "true" in hdfs-site.xml and that seemed to have solved the issue. Of course we are using HDFS which does support append. I think the actual configuration Spark should check

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

Hi Arijit, Have you find a solution for this? I'm facing the same problem in Spark 1.6.1, but here the error happens only a few times, so our hdfs does support append. This is what I can see in the logs: 2016-11-17 13:43:20,012 ERROR [BatchedWriteAheadLog Writer] WriteAheadLogManager for Thread:

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

Thanks TD. Is "hdfs.append.support" a standard configuration? I see a seemingly equivalent configuration "dfs.support.append" that is used in our version of HDFS. In case we want to use a pseudo file-system (like S3) which does not support append what are our options? I am not familiar with

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

For WAL in Spark to work with HDFS, the HDFS version you are running must support file appends. Contact your HDFS package/installation provider to figure out whether this is supported by your HDFS installation. On Mon, Nov 7, 2016 at 2:04 PM, Arijit wrote: > Hello All, > > >

Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

Hello All, We are using Spark 1.6.2 with WAL enabled and encountering data loss when the following exception/warning happens. We are using HDFS as our checkpoint directory. Questions are: 1. Is this a bug in Spark or issue with our configuration? Source looks like the following. Which