[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-09-20 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621742#comment-16621742
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

tillrohrmann commented on issue #5530: [FLINK-8660] Enable the user to provide 
custom HAServices implementation
URL: https://github.com/apache/flink/pull/5530#issuecomment-423115523
 
 
   Merged via 3bc7bc2e80af1e277ffb4906e6aca416a9de954b and 
324d7bed226cb707512879842c33788f7f715ff2


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Assignee: Krzysztof Białek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-09-20 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621743#comment-16621743
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

tillrohrmann closed pull request #5530: [FLINK-8660] Enable the user to provide 
custom HAServices implementation
URL: https://github.com/apache/flink/pull/5530
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/ops/jobmanager_high_availability.md 
b/docs/ops/jobmanager_high_availability.md
index a99455849c4..669919e6618 100644
--- a/docs/ops/jobmanager_high_availability.md
+++ b/docs/ops/jobmanager_high_availability.md
@@ -65,6 +65,7 @@ By default, the job manager will pick a *random port* for 
inter process communic
 In order to start an HA-cluster add the following configuration keys to 
`conf/flink-conf.yaml`:
 
 - **high-availability mode** (required): The *high-availability mode* has to 
be set in `conf/flink-conf.yaml` to *zookeeper* in order to enable high 
availability mode.
+Alternatively this option can be set to FQN of factory class Flink should use 
to create HighAvailabilityServices instance. 
 
   high-availability: zookeeper
 
diff --git 
a/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java
 
b/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java
index 8ef605bb328..51c21053df9 100644
--- 
a/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java
+++ 
b/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java
@@ -39,13 +39,14 @@
 * Defines high-availability mode used for the cluster execution.
 * A value of "NONE" signals no highly available setup.
 * To enable high-availability, set this mode to "ZOOKEEPER".
+* Can also be set to FQN of HighAvailability factory class.
 */
public static final ConfigOption HA_MODE =
key("high-availability")
.defaultValue("NONE")
.withDeprecatedKeys("recovery.mode")
.withDescription("Defines high-availability mode used 
for the cluster execution." +
-   " To enable high-availability, set this mode to 
\"ZOOKEEPER\".");
+   " To enable high-availability, set this mode to 
\"ZOOKEEPER\" or specify FQN of factory class.");
 
/**
 * The ID of the Flink cluster, used to separate multiple Flink clusters
@@ -67,7 +68,6 @@

.withDeprecatedKeys("high-availability.zookeeper.storageDir", 
"recovery.zookeeper.storageDir")
.withDescription("File system path (URI) where Flink 
persists metadata in high-availability setups.");
 
-
// 

//  Recovery Options
// 

diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
index a21c7d61fcb..77b617bbdcc 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
@@ -87,14 +87,10 @@
 *  thrown if the (distributed) file storage cannot be 
created
 */
public static BlobStoreService createBlobStoreFromConfig(Configuration 
config) throws IOException {
-   HighAvailabilityMode highAvailabilityMode = 
HighAvailabilityMode.fromConfig(config);
-
-   if (highAvailabilityMode == HighAvailabilityMode.NONE) {
-   return new VoidBlobStore();
-   } else if (highAvailabilityMode == 
HighAvailabilityMode.ZOOKEEPER) {
+   if 
(HighAvailabilityMode.isHighAvailabilityModeActivated(config)) {
return createFileSystemBlobStore(config);
} else {
-   throw new IllegalConfigurationException("Unexpected 
high availability mode '" + highAvailabilityMode + "'.");
+   return new VoidBlobStore();
}
}
 
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/HighAvailabilityServicesFactory.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/HighAvailabilityServicesFactory.java
new file mode 100644
index 000..a0e897739c1
--- /dev/null
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/HighAvailabilityServicesFactory.java
@@ -0,0

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-09-07 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607352#comment-16607352
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

tillrohrmann commented on issue #5530: [FLINK-8660] Enable the user to provide 
custom HAServices implementation
URL: https://github.com/apache/flink/pull/5530#issuecomment-419501245
 
 
   Yes I think we should try to get this PR into 1.7 since it allows to 
implement your own `HighAvailabilityServices`. I'll try to find some time next 
week to take a look what the current state is.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Assignee: Krzysztof Białek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-09-02 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601212#comment-16601212
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Xeli commented on issue #5530: [FLINK-8660] Enable the user to provide custom 
HAServices implementation
URL: https://github.com/apache/flink/pull/5530#issuecomment-417931651
 
 
   I am in the process of building a HAService for our kubernetes set up 
without having to use zookeeper (https://stackoverflow.com/q/52104759/988324) 
and this PR would be very helpful.
   
   Would it be possible to merge this into 1.7? (or 1.6.1? :D)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Assignee: Krzysztof Białek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-03-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392908#comment-16392908
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5530
  
I think this looks quite good now.

Since 1.5 feature freeze is over for a bit, would suggest to merge this 
into master (1.6-SNAPSHOT).

Looking at the code (this PR and the high availability configuration in 
general), I would suggest to work towards the following design in the future:

  - We don't have specific high availability enums and setting, everything 
is just an implementation of HAServices. Some HAServices are in fact not highly 
available (maybe we need to find a better name for HAServices).
  - There are factories for all HAServices
  - We simply have "none" and "zookeeper" as aliases for the respective 
factories.

That mirrors the design of he state backends and their configuration, which 
is nice, because we never need to do special checks like 
"isHighAvailabilityEnabled".


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375750#comment-16375750
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5530
  
You could rebase on the latest master and force push to retrigger the build.


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375607#comment-16375607
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user kbialek commented on the issue:

https://github.com/apache/flink/pull/5530
  
Looks like build process was terminated by kernel. Can someone retrigger it?



> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375562#comment-16375562
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user kbialek commented on the issue:

https://github.com/apache/flink/pull/5530
  
Now uses high-availability configuration property can be set either to 
none, zookeeper or FQN of factory class


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375561#comment-16375561
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

GitHub user kbialek reopened a pull request:

https://github.com/apache/flink/pull/5530

[FLINK-8660] Enable the user to provide custom HAServices implementation

## What is the purpose of the change
This pull request makes it possible to use alternative (to ZK) HA backend 
implementation

## Brief change log
* Added HighAvailabilityServicesFactory interface
* Added new option HighAvailabilityMode.CUSTOM
* Added new configuration property high-availability.factoryClass
* HighAvailabilityServicesUtils uses configured factory to instantiate 
HighAvailabilityServices
* Updated docs

## Verifying this change
This change added tests and can be verified as follows:
* Added new test class HighAvailabilityServicesUtilsTest
* Updated test class HighAvailabilityModeTest
* Manually verified by using own HA backend implementation

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes, if this feature is 
used, otherwise no
  - The S3 file system connector: no

## Documentation
  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? docs and JavaDocs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kbialek/flink FLINK-8660

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5530


commit 494bb2f7175bb174ed2a7139495af05891012498
Author: Krzysztof Białek 
Date:   2018-02-17T15:11:50Z

Implemented

commit 077102571ae58987fafbf00ba536f1512c700e1b
Author: Krzysztof Białek 
Date:   2018-02-17T16:45:47Z

Create BlobStorage for any HA backend

commit 57e52b70700dc1b3a4f05113ae6aa594112eead7
Author: Krzysztof Białek 
Date:   2018-02-17T16:46:02Z

HighAvailabilityServicesFactory may throw exceptions

commit 9a08884ee05f8a8bf8e03b7fdd3b6d148530c40b
Author: Krzysztof Białek 
Date:   2018-02-19T21:12:59Z

Docs

commit 0afc341077347e32669330ea66d8518d9a3133ef
Author: zentol 
Date:   2018-02-07T08:52:23Z

[FLINK-8574][travis] Add timestamp to logging messages

This closes #5419.

commit e2f1650ea251ab7a7f6090cda819e2e2481ffdf5
Author: zentol 
Date:   2018-02-13T09:31:31Z

[FLINK-8621][prometheus][tests] Remove 
endpointIsUnavailableAfterReporterIsClosed()

The test is inherently unstable as it will always fail if any other
server is started on the port between the closing of the reporter and
the polling of metrics.

This closes #5473.

commit 2db296973ec2aa648246279a2e55cca87432f829
Author: zentol 
Date:   2018-02-13T09:33:03Z

[hotfix][prometheus] Document internal usage of 
CollectorRegistry.defaultRegistry

It appeared as if the HTTPServer wasn't actually doing anything, but it
internally accessed the singleton registry that we also access to
register metrics.

commit 6fb2c89f7cfa5b3d05c683777b76d257db4770c5
Author: zentol 
Date:   2018-02-13T09:33:31Z

[hotfix][prometheus][tests] Add utility for generating port ranges

commit 7441b16a4eeb10633aca4b9c89629ee92a182092
Author: Aljoscha Krettek 
Date:   2018-02-14T13:48:22Z

[FLINK-8600] Allow disabling truncate() check in BucketingSink

The test was failing when using PrestoS3FileSystem because it doesn't
use an absolute/qualified path.

commit 9f5e835e845901fd1183b390c510851c2ca17b03
Author: Till Rohrmann 
Date:   2018-02-19T11:00:08Z

[FLINK-8698] [flip6] Use Flip6LocalStreamEnvironment instead of 
LocalStreamEnvironment

commit 9b135941c5295af4db074aa50d5bb881a6a52feb
Author: Till Rohrmann 
Date:   2018-02-19T11:41:41Z

[FLINK-8698] [flip6] Let LocalExecutor use Flip-6 MiniCluster

This closes #5524.

commit 4596c656c79829e2c73aee252ce13c0cb5bc9b7a
Author: Till Rohrmann 
Date:   2018-02-20T08:00:31Z

[FLINK-8709] [tests] Harden 
SlotPoolRpcTest.testCancelSlotAllocationWithoutResourceManager

commit 2d9db7204d3c4b012600416038679941241f1088
Author: zentol 
Date:   2018-02-20T16:31:05Z

[hotfix][docs] Update configuration docs

commit f75b763a40dcb6a1b1b06a7312d5ee7b67f19a5c
Author: Krzysztof Białek 
Date:

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-20 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370487#comment-16370487
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user kbialek commented on the issue:

https://github.com/apache/flink/pull/5530
  
Closing without merge, because of:
1. Test failure
2. I'd like to implement Stephan's idea


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-20 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370489#comment-16370489
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user kbialek closed the pull request at:

https://github.com/apache/flink/pull/5530


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-20 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370484#comment-16370484
 ] 

Krzysztof Białek commented on FLINK-8660:
-

[~StephanEwen] suggested to simplify configuration 
[https://github.com/apache/flink/pull/5530#issuecomment-366921300]
{quote}I would suggest to not split the config options between 
{{high-availability}} and factory that is only used in _CUSTOM_ mode, but use 
the {{high-availability}} option for both. The option describes the high 
availability services factory, with _NONE_ and _ZOOKEEPER_ as special 
constants/shortcuts.

Have a look at the {{StateBackendLoader}} class for an example of that.
[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L92]

 
{quote}
 

> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-20 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369853#comment-16369853
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5530
  
I think this is a nice approach.

I would suggest to not split the config options between `high-availability` 
and factory that is only used in *CUSTOM* mode, but use the `high-availability` 
option for both. The option describes the high availability services factory, 
with *NONE* and *ZOOKEEPER* as special constants/shortcuts.

Have a look at the `StateBackendLoader` class for an example of that.

https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L92


> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-19 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369537#comment-16369537
 ] 

ASF GitHub Bot commented on FLINK-8660:
---

GitHub user kbialek opened a pull request:

https://github.com/apache/flink/pull/5530

[FLINK-8660] Enable the user to provide custom HAServices implementation

## What is the purpose of the change
This pull request makes it possible to use alternative (to ZK) HA backend 
implementation

## Brief change log
* Added HighAvailabilityServicesFactory interface
* Added new option HighAvailabilityMode.CUSTOM
* Added new configuration property high-availability.factoryClass
* HighAvailabilityServicesUtils uses configured factory to instantiate 
HighAvailabilityServices
* Updated docs

## Verifying this change
This change added tests and can be verified as follows:
* Added new test class HighAvailabilityServicesUtilsTest
* Updated test class HighAvailabilityModeTest
* Manually verified by using own HA backend implementation

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes, if this feature is 
used, otherwise no
  - The S3 file system connector: no

## Documentation
  - Does this pull request introduce a new feature? yes
  - If yes, how is the feature documented? docs and JavaDocs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kbialek/flink FLINK-8660

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5530


commit 494bb2f7175bb174ed2a7139495af05891012498
Author: Krzysztof Białek 
Date:   2018-02-17T15:11:50Z

Implemented

commit 077102571ae58987fafbf00ba536f1512c700e1b
Author: Krzysztof Białek 
Date:   2018-02-17T16:45:47Z

Create BlobStorage for any HA backend

commit 57e52b70700dc1b3a4f05113ae6aa594112eead7
Author: Krzysztof Białek 
Date:   2018-02-17T16:46:02Z

HighAvailabilityServicesFactory may throw exceptions

commit 9a08884ee05f8a8bf8e03b7fdd3b6d148530c40b
Author: Krzysztof Białek 
Date:   2018-02-19T21:12:59Z

Docs




> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8660) Enable the user to provide custom HAServices implementation

2018-02-15 Thread Till Rohrmann (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365244#comment-16365244
 ] 

Till Rohrmann commented on FLINK-8660:
--

This sounds like a really good idea [~kbialek]. The plan sounds good to me and 
I think we should proceed with it.

> Enable the user to provide custom HAServices implementation 
> 
>
> Key: FLINK-8660
> URL: https://issues.apache.org/jira/browse/FLINK-8660
> Project: Flink
>  Issue Type: Improvement
>  Components: Cluster Management, Configuration, Distributed 
> Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Krzysztof Białek
>Priority: Major
> Fix For: 1.6.0
>
>
> At the moment Flink uses ZooKeeper as HA backend.
> The goal of this improvement is to make Flink supporting more HA backends, 
> also maintained as independent projects.
> The following changes are required to achieve it:
>  # Add {{HighAvailabilityServicesFactory}} interface
>  # Add new option {{HighAvailabilityMode.CUSTOM}}
>  # Add new configuration property {{high-availability.factoryClass}}
>  # Use the factory in {{HighAvailabilityServicesUtils}} to instantiate  
> {{HighAvailabilityServices}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)