GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/1227
[FLINK-2805] [blobmanager] Write JARs to file state backend for recovery
This is a follow up to #1153. I've taken two changes from #1153 for
convenience. Other than that, this is independent.
When running the `BlobServer` in `RecoveryMode#ZOOKEEPER`, this will upload
the JARs to the configured file system backend (e.g. HDFS). **Important**: it
introduces a hard dependency to have a configured file state backend when
running the blob server with recovery. This is in line with #1153.
This JAR copying only happens on the server side, e.g. the client uploads
to the server and the server uploads it to the state backend. Same when
requesting a locally non-existing blob: the client requests from the server and
the server downloads if not available and then answers the client.
There are other ways to implement this, but this one was minimally invasive
and fully circumvents any Akka actor threads for downloading/uploading. A more
invasive change could allow to directly interact with the state backend on the
client side [task manager] as well. This would spread the load better across
the cluster in case of a DFS and save unnecessary network transfers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/flink hdfs_jars-2805
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1227.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1227
----
commit 68a7ac4abab15dc2ef45546de6a94c27129534dc
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-10-05T12:30:46Z
[FLINK-2805] Apply RecoveryMode and ConfigConstants changes from #1153
commit 00ea6b02f15f00486c9541af08b8197c41dd94f7
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-10-05T08:05:05Z
[FLINK-2805] [blobmanager] Write JARs to file state backend for recovery
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
