Questions on Python support with Spark

those. Thanks, Arijit

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

n Spark should check is "dfs.support.append". I believe failure is intermittent since in most cases a new file is created to store the block addition event. I need to look into the code again to see when these files are created new and when they are appended. Thanks, Arijit ___

Spark streaming data loss due to timeout in writing BlockAdditionEvent to WAL by the driver

(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) We tried with increasing the timeout to 60 seconds but could not eliminate the issue completely. Requesting suggestions on what would be the recourse to stop this data bleeding. Thanks, Arijit

Re: Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

am not familiar with the code yet but is it possible to generate a new file whenever conflict of this sort happens? Thanks again, Arijit From: Tathagata Das Sent: Monday, November 7, 2016 7:59:06 PM To: Arijit Cc: user@spark.apache.org Subject: Re: Spark St

Spark Streaming Data loss on failure to write BlockAdditionEvent failure to WAL

.11:42316 (size: 283.1 KB, free: 2.6 GB) I am sure Spark Streaming is not expected to lose data when WAL is enabled. So what are we doing wrong here? Thanks, Arijit

Fw: Issue with Spark Streaming with checkpointing in Spark 2.0

Resending, not sure if had sent to user@spark.apache.org earlier. Thanks, Arijit From: Arijit Sent: Friday, October 7, 2016 6:06 PM To: user@spark.apache.org Subject: Issue with Spark Streaming with checkpointing in Spark 2.0 In a Spark Streaming sample