[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164998
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/filesystem/FileSystemStateStorageHelper.java
 ---
@@ -77,4 +83,18 @@ public FileSystemStateStorageHelper(Path rootPath, 
String prefix) throws IOExcep
private Path getNewFilePath() {
return new Path(rootPath, FileUtils.getRandomFilename(prefix));
}
+
+   @Override
+   public void closeAndCleanupAllData() {
+   try {
+   fs.delete(rootPath, true);
--- End diff --

We cannot simply delete the rootPath, because it's 
`HighAvailabilityOptions.HA_STORAGE_PATH`. Thus, we would also delete files 
from other components in this directory (e.g. the `BlobServer`). I think the 
`FileSystemStateStorageHelper` has to create a sub directory for the state to 
store. That directory, we can easily delete.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164895
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java
 ---
@@ -102,6 +104,8 @@
/** Store for arbitrary blobs */
private final BlobStoreService blobStoreService;
 
+   private RetrievableStateStorageService stateStorage;
--- End diff --

should be `final`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164914
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java
 ---
@@ -190,6 +198,18 @@ public void closeAndCleanupAllData() throws Exception {
exception = t;
}
 
+   try {
+   stateStorage.closeAndCleanupAllData();
+   } catch (Throwable t) {
+   exception = t;
--- End diff --

add as suppressed exception using `ExceptionUtils.firstOrSuppressed`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164958
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java 
---
@@ -244,6 +244,7 @@ public static ZooKeeperLeaderElectionService 
createLeaderElectionService(
 * @return {@link ZooKeeperSubmittedJobGraphStore} instance
 * @throws Exception if the submitted job graph store cannot be created
 */
+   @Deprecated
--- End diff --

Can we remove this method? If not, then at least write why this method was 
deprecated using the `@deprecated` tag.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164937
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/filesystem/FileSystemStateStorageHelper.java
 ---
@@ -77,4 +83,18 @@ public FileSystemStateStorageHelper(Path rootPath, 
String prefix) throws IOExcep
private Path getNewFilePath() {
return new Path(rootPath, FileUtils.getRandomFilename(prefix));
}
+
+   @Override
+   public void closeAndCleanupAllData() {
+   try {
+   fs.delete(rootPath, true);
+   } catch (Exception e) {
+   LOG.error("Failed to clean up state storage 
directory.", e);
--- End diff --

I would not catch this exception


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-07-01 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4204#discussion_r125164898
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java
 ---
@@ -113,6 +117,7 @@ public ZooKeeperHaServices(
this.runningJobsRegistry = new 
ZooKeeperRunningJobsRegistry(client, configuration);
 
this.blobStoreService = checkNotNull(blobStoreService);
+   this.stateStorage = null;
--- End diff --

let's eagerly initialize this field


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...

2017-06-27 Thread zjureel 
GitHub user zjureel opened a pull request:

https://github.com/apache/flink/pull/4204

[FLINK-6522] Add ZooKeeper cleanup logic to ZooKeeperHaServices

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjureel/flink FLINK-6522

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4204


commit b816aa3a2a3d4d1377caccdcbeba91073e89af75
Author: zjureel 
Date:   2017-06-28T02:12:27Z

clean up all data which is stored by RetrievableStateStorageHelper

commit 03a2f39e051594cb07017ef2613eb7873b56d118
Author: zjureel 
Date:   2017-06-28T04:42:30Z

clean up all data which is stored in zookeeper




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---