GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/22882
[SPARK-25871][STREAMING][WIP] Don't use EC for streaming WAL
## What changes were proposed in this pull request?
The write ahead log expects to be able to call hflush, but that is a no-op
when writing to a file with hdfs erasure coding. So ensure that file is always
written with replication instead, regardless of filesystem defaults.
Note this is a WIP on top of changes from
https://github.com/apache/spark/pull/22881. The only new change here is
https://github.com/apache/spark/commit/98204e6bcb840f1a47e1a3bd73da5fd7c9b22bcd
## How was this patch tested?
None yet. I'm posting this mostly to make it visible, as it was trivial on
top of https://github.com/apache/spark/pull/22881
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-25871
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22882.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22882
----
commit 005ee5494acd3d9f0721ad24ba3700d8905e2e26
Author: Imran Rashid <irashid@...>
Date: 2018-10-26T19:03:43Z
[SPARK-25855][CORE][STREAMING] Don't use HDFS EC for event logs and WAL
hdfs erasure coding doesn't support hflush(), hsync(), or append(),
which doesn't work well for event logs and the WAL, so be sure we never
use it for those files, regardless of the configuration of hdfs.
commit 04b968a0223e195f1c7e6d6684274bd7f8484069
Author: Imran Rashid <irashid@...>
Date: 2018-10-26T20:22:11Z
fix
commit 8a9392c875b9b2aec048940a8ae7d03529bfc641
Author: Imran Rashid <irashid@...>
Date: 2018-10-29T15:56:20Z
make it configurable
commit cd28e61fe9232927ea66b3beb4af5c5d699bb6d3
Author: Imran Rashid <irashid@...>
Date: 2018-10-29T20:09:28Z
remove changes for WAL
commit 98204e6bcb840f1a47e1a3bd73da5fd7c9b22bcd
Author: Imran Rashid <irashid@...>
Date: 2018-10-29T20:12:14Z
Add back changes for the WAL
This reverts commit cd28e61fe9232927ea66b3beb4af5c5d699bb6d3.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
