Re: [Structured Streaming SPARK-23966] Why non-atomic rename is problem in State Store ?

2018-10-02 Thread chandan prakash
, Chandan On Mon, Oct 1, 2018 at 5:37 PM Steve Loughran wrote: > > > On 11 Aug 2018, at 17:33, chandan prakash > wrote: > > Hi All, > I was going through this pull request about new CheckpointFileManager > abstraction in structured streaming coming in 2.4 : > https://is

Re: [Structured Streaming SPARK-23966] Why non-atomic rename is problem in State Store ?

2018-09-30 Thread chandan prakash
Anyone who can clear doubts on the questions asked here ? Regards, Chandan On Sat, Aug 11, 2018 at 10:03 PM chandan prakash wrote: > Hi All, > I was going through this pull request about new CheckpointFileManager > abstraction in structured streaming coming in 2.4

[Structured Streaming SPARK-23966] Why non-atomic rename is problem in State Store ?

2018-08-11 Thread chandan prakash
Hi All, I was going through this pull request about new CheckpointFileManager abstraction in structured streaming coming in 2.4 : https://issues.apache.org/jira/browse/SPARK-23966 https://github.com/apache/spark/pull/21048 I went through the code in detail and found it will indtroduce a very nice