Gabor Somogyi created SPARK-23438: ------------------------------------- Summary: DStreams could lose blocks with WAL enabled when driver crashes Key: SPARK-23438 URL: https://issues.apache.org/jira/browse/SPARK-23438 Project: Spark Issue Type: Bug Components: DStreams Affects Versions: 1.6.0 Reporter: Gabor Somogyi
There is a race condition introduced in SPARK-11141 which could cause data loss. This affects all versions since 1.6.0. Problematic situation: # Start streaming job with 2 receivers with WAL enabled. # Receiver 1 receives a block and does the following ** Writes a BlockAdditionEvent into WAL ** Puts the block into it's received block queue with ID 1 # Receiver 2 receives a block and does the following ** Writes a BlockAdditionEvent into WAL # Spark allocates all blocks from it's received block queue and writes AllocatedBlocks(IDs=(1)) into WAL # Driver crashes # New Driver recovers from WAL # Realise block with ID 2 never processed -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org