[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709844#comment-16709844
]
ASF GitHub Bot commented on FLINK-6065:
---
hzyuemeng1 opened a new pull request #7244: [FLINK-6065][es-connector] Make
TransportClient for ES5 pluggable
URL: https://github.com/apache/flink/pull/7244
## What is the purpose of the change
*(For example: This pull request makes task deployment go through the blob
server, rather than through RPC. That way we avoid re-transferring them on each
deployment (during recovery).)*
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a
persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(100MB)*
- *Extended integration test for recovery after master (JobManager)
failure*
- *Added test that validates that TaskInfo is transferred only once across
recoveries*
- *Manually verified the change by running a 4 node cluser with 2
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one
JobManager and two TaskManagers during the execution, verifying that recovery
happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no /
don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>Priority: Major
> Labels: pull-request-available
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708147#comment-16708147
]
ASF GitHub Bot commented on FLINK-6065:
---
hzyuemeng1 commented on issue #3934: [FLINK-6065] Add initClient method to
ElasticsearchApiCallBridge
URL: https://github.com/apache/flink/pull/3934#issuecomment-443957496
@tzulitai ,is anyone still work for this issue.
If not, can I do this job
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>Priority: Major
> Labels: pull-request-available
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295316#comment-16295316
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user sschaef closed the pull request at:
https://github.com/apache/flink/pull/3934
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056958#comment-16056958
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3934
Hi @sschaef! I was wondering if you would like to finish this work? Either
way, please let me know :)
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017234#comment-16017234
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3934
Answer to some of your other questions:
1. I don't think the Apache ICLA is strictly required, unless you're
invited as a Committer of the project.
2. The tests should be added under `src/test/...` directory of the module.
I think the already-existing IT tests for the ES sinks already cover the
changed functionality here. Since this isn't a very complicated feature, for
new tests, at the very most I would suggest perhaps to add unit tests that test
the factory is correctly respected during the init flow.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017230#comment-16017230
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/3934#discussion_r117455309
--- Diff:
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchApiCallBridge.java
---
@@ -40,12 +43,27 @@
/**
* Creates an Elasticsearch {@link Client}.
*
+* In comparison to {@link initClient}, this method creates a default
{@link Client}.
+*
* @param clientConfig The configuration to use when constructing the
client.
* @return The created client.
*/
Client createClient(Map clientConfig);
/**
+* Initializes an Elasticsearch {@link Client}.
+*
+* A {@link Settings} object is created, which is passed to {@link
mapper} in order to allow the creation of a
+* {@link TransportClient}. {@link createClient} creates and
initializes a default client for cases where the
+* implementation doesn't matter.
+*
+* @param clientConfig The configuration to use when constructing the
client.
+* @param mapper Receives a {@link Settings} object that can be used to
create a {@link TransportClient}.
+* @return The initialized client that has been created by {@link
mapper}.
+*/
+ Client initClient(Map clientConfig, Function mapper);
--- End diff --
As for the defaults, a default `DefaultTransportClientFactory` that
replaces the previous hardcodes can be provided to this method.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017224#comment-16017224
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/3934#discussion_r117451847
--- Diff:
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchApiCallBridge.java
---
@@ -40,12 +43,27 @@
/**
* Creates an Elasticsearch {@link Client}.
*
+* In comparison to {@link initClient}, this method creates a default
{@link Client}.
+*
* @param clientConfig The configuration to use when constructing the
client.
* @return The created client.
*/
Client createClient(Map clientConfig);
/**
+* Initializes an Elasticsearch {@link Client}.
+*
+* A {@link Settings} object is created, which is passed to {@link
mapper} in order to allow the creation of a
+* {@link TransportClient}. {@link createClient} creates and
initializes a default client for cases where the
+* implementation doesn't matter.
+*
+* @param clientConfig The configuration to use when constructing the
client.
+* @param mapper Receives a {@link Settings} object that can be used to
create a {@link TransportClient}.
+* @return The initialized client that has been created by {@link
mapper}.
+*/
+ Client initClient(Map clientConfig, Function mapper);
--- End diff --
I wonder if a `TransportClientFactory` would be more intuitive here? That
would also solve the Java 8 problem.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017226#comment-16017226
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/3934#discussion_r117452130
--- Diff:
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchApiCallBridge.java
---
@@ -40,12 +43,27 @@
/**
* Creates an Elasticsearch {@link Client}.
*
+* In comparison to {@link initClient}, this method creates a default
{@link Client}.
--- End diff --
I don't think the `{@link initClient}` Javadoc link have the correct
signature, does it?
For example, `{@link initClient}` should be `{@link #initClient(Map, Function)}` to actually link to that method.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017225#comment-16017225
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/3934#discussion_r117452728
--- Diff:
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchApiCallBridge.java
---
@@ -40,12 +43,27 @@
/**
* Creates an Elasticsearch {@link Client}.
*
+* In comparison to {@link initClient}, this method creates a default
{@link Client}.
+*
* @param clientConfig The configuration to use when constructing the
client.
* @return The created client.
*/
Client createClient(Map clientConfig);
--- End diff --
The `ElasticsearchApiCallBridge` actually isn't a user-facing class, only
an interface for our version-specific implementations. So I think it is ok to
alter the interface freely.
In that case, it shouldn't be necessary to have a new method.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014957#comment-16014957
]
ASF GitHub Bot commented on FLINK-6065:
---
Github user sschaef commented on the issue:
https://github.com/apache/flink/pull/3934
The Java7 run failed because my solution uses Java8 features. I guess I
have to find another way to fix this issue then.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6065) Make TransportClient for ES5 pluggable
[
https://issues.apache.org/jira/browse/FLINK-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014817#comment-16014817
]
ASF GitHub Bot commented on FLINK-6065:
---
GitHub user sschaef opened a pull request:
https://github.com/apache/flink/pull/3934
[FLINK-6065] Add initClient method to ElasticsearchApiCallBridge
This adds the method `initClient` to `ElasticsearchApiCallBridge` in
order to resolve FLINK-6065. This new method takes as argument a
function that can create a `TransportClient`. This is required in order
to not hardcode the `TransportClient` in the implementation of
`createClient`. `createClient` continues to exist in order to allow
backwards compatibility.
No tests provided because I couldn't find any existing test class that
tests the implementation of `ElasticsearchApiCallBridge`.
This is my first contribution, I haven't signed the ICLA yet (which I will
do next). I didn't provide any tests yet because I had no idea how the
functionality should be tested or where a test class should be added. If you
would like to see tests for the changes, please tell me.
The change fixes the ticket but maybe you would like to see a different way
on how to fix the issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sschaef/flink flink-6065
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3934.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3934
commit 2815b0c9ca407838611ba60b9c35a0ac550005e3
Author: Simon Schäfer
Date: 2017-05-17T20:24:14Z
[FLINK-6065] Add initClient method to ElasticsearchApiCallBridge
This adds the method `initClient` to `ElasticsearchApiCallBridge` in
order to resolve FLINK-6065. This new method takes as argument a
function that can create a `TransportClient`. This is required in order
to not hardcode the `TransportClient` in the implementation of
`createClient`. `createClient` continues to exist in order to allow
backwards compatibility.
No tests provided because I couldn't find any existing test class that
tests the implementation of `ElasticsearchApiCallBridge`.
> Make TransportClient for ES5 pluggable
> --
>
> Key: FLINK-6065
> URL: https://issues.apache.org/jira/browse/FLINK-6065
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector, Streaming Connectors
>Reporter: Robert Metzger
>
> This JIRA is based on a user request:
> http://stackoverflow.com/questions/42807454/flink-xpack-elasticsearch-5-elasticsearchsecurityexception-missing-autentication?noredirect=1#comment72728053_42807454
> Currently, in the {{Elasticsearch5ApiCallBridge}} the
> {{PreBuiltTransportClient}} is hardcoded. It would be nice to make this
> client pluggable to allow using other clients such as the
> {{PreBuiltXPackTransportClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
