RE: Using Flink's Kubernetes API inside Java
Thanks Roman and Yang, I understand. I’ll have a look and ask on the developer
list depending on what I find.
Regards,
Alexis.
From: Yang Wang
Sent: Mittwoch, 7. Juli 2021 05:14
To: ro...@apache.org
Cc: Alexis Sarda-Espinosa ;
user@flink.apache.org
Subject: Re: Using Flink's Kubernetes API inside Java
Hi Alexis,
I used to create a ticket[1] to make the ApplicationDeployer interface public.
However, the community is very careful to add new public interfaces.
Because it will make the maintenance more difficult. AFAIK, the
ApplicationDeployer is stable and it is a very basic requirement for
Application mode.
I do not see an obvious reason to change it in the future.
Unfortunately, Flink does not have a common interface for deploying a session
cluster. I think you could have your own SessionClusterDeployer, it could be
implemented as follows.
final ClusterClientFactory clientFactory =
clientServiceLoader.getClusterClientFactory(configuration);
try (final ClusterDescriptor clusterDescriptor =
clientFactory.createClusterDescriptor(configuration)) {
final ClusterSpecification clusterSpecification =
clientFactory.getClusterSpecification(configuration);
clusterDescriptor.deploySessionCluster(clusterSpecification);
By setting the DeploymentOptions.TARGET to "kubernetes-session", you could
activate the K8s descriptor.
[1]. https://issues.apache.org/jira/browse/FLINK-21807
Best,
Yang
Roman Khachatryan mailto:ro...@apache.org>> 于2021年7月7日周三
上午12:18写道：
Hi Alexis,
KubernetesSessionCli provides a similar functionality IIUC but it's
also marked as @Internal (so it likely will change in the future; the
REST APIs it uses aren't likely to change, but I guess it doesn't help
as you'd like some helper classes.).
I think it's a good idea to ask this question on a dev mailing list.
Regards,
Roman
On Fri, Jul 2, 2021 at 11:19 PM Alexis Sarda-Espinosa
mailto:alexis.sarda-espin...@microfocus.com>>
wrote:
>
> Hi Roman,
>
> In the operator I mentioned I see logic like the one here:
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/a60a9826d4bcdaa4f23cf296d95954b9f6f328c3/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkApplicationController.java#L169
>
> For instance, DefaultClusterClientServiceLoader is annotated with @Internal:
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/client/deployment/DefaultClusterClientServiceLoader.html
>
> And just like ApplicationClusterDeployer is in charge of Application Mode
> (according to the Javadoc), I was wondering if there's something similar for
> Session Mode.
>
> Perhaps I should subscribe to the developer mailing list? Although I guess
> that's part of the question, whether those classes count as User API or
> Developer API.
>
> I took a quick glance at the operator you mentioned, but I'm hoping I can
> make use of Flink's new support for pod templates to make it as similar as
> possible to a native Deployment resource.
>
> Regards,
> Alexis.
>
> From: Roman Khachatryan mailto:ro...@apache.org>>
> Sent: Friday, July 2, 2021 9:19 PM
> To: Alexis Sarda-Espinosa
> mailto:alexis.sarda-espin...@microfocus.com>>;
> Yang Wang mailto:danrtsey...@gmail.com>>
> Cc: user@flink.apache.org<mailto:user@flink.apache.org>
> mailto:user@flink.apache.org>>
> Subject: Re: Using Flink's Kubernetes API inside Java
>
> Hi Alexis,
>
> Have you looked at flink-on-k8s-operator [1]?
> It seems to have the functionality you need:
> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569
>
> I couldn't find many Flink-specific classes in the operator you
> mentioned, but in general classes annotated with Public and
> PublicEvolving are unlikely to change if that's your concern.
>
> Also pulling in Yang Wang.
>
> [1]
> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/
>
>
> Regards,
> Roman
>
>
> On Thu, Jul 1, 2021 at 7:49 PM Alexis Sarda-Espinosa
> mailto:alexis.sarda-espin...@microfocus.com>>
> wrote:
> >
> > Hello everyone,
> >
> > I'm testing a custom Kubernetes operator that should fulfill some specific
> > requirements I have for Flink. I know of this WIP project:
> > https://github.com/wangyang0918/flink-native-k8s-operator
> >
> > I can see that it uses some classes that aren't publicly documented, and I
> > believe it doesn't cover Flink K8s sessions, which I would like to use.
> >
> > Is there something I could use for Flink K8s sessions? And is it ok if I
> > use these classes knowing that I might need adjustments for future Flink
> > versions?
> >
> > Regards,
> > Alexis.
> >
Re: Using Flink's Kubernetes API inside Java
Hi Alexis,
I used to create a ticket[1] to make the ApplicationDeployer interface
public. However, the community is very careful to add new public interfaces.
Because it will make the maintenance more difficult. AFAIK, the
ApplicationDeployer is stable and it is a very basic requirement for
Application mode.
I do not see an obvious reason to change it in the future.
Unfortunately, Flink does not have a common interface for deploying a
session cluster. I think you could have your own SessionClusterDeployer, it
could be
implemented as follows.
final ClusterClientFactory clientFactory =
clientServiceLoader.getClusterClientFactory(configuration);
try (final ClusterDescriptor clusterDescriptor =
clientFactory.createClusterDescriptor(configuration)) {
final ClusterSpecification clusterSpecification =
clientFactory.getClusterSpecification(configuration);
clusterDescriptor.deploySessionCluster(clusterSpecification);
By setting the DeploymentOptions.TARGET to "kubernetes-session", you could
activate the K8s descriptor.
[1]. https://issues.apache.org/jira/browse/FLINK-21807
Best,
Yang
Roman Khachatryan 于2021年7月7日周三 上午12:18写道：
> Hi Alexis,
>
> KubernetesSessionCli provides a similar functionality IIUC but it's
> also marked as @Internal (so it likely will change in the future; the
> REST APIs it uses aren't likely to change, but I guess it doesn't help
> as you'd like some helper classes.).
> I think it's a good idea to ask this question on a dev mailing list.
>
>
> Regards,
> Roman
>
> On Fri, Jul 2, 2021 at 11:19 PM Alexis Sarda-Espinosa
> wrote:
> >
> > Hi Roman,
> >
> > In the operator I mentioned I see logic like the one here:
> https://github.com/wangyang0918/flink-native-k8s-operator/blob/a60a9826d4bcdaa4f23cf296d95954b9f6f328c3/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkApplicationController.java#L169
> >
> > For instance, DefaultClusterClientServiceLoader is annotated with
> @Internal:
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/client/deployment/DefaultClusterClientServiceLoader.html
> >
> > And just like ApplicationClusterDeployer is in charge of Application
> Mode (according to the Javadoc), I was wondering if there's something
> similar for Session Mode.
> >
> > Perhaps I should subscribe to the developer mailing list? Although I
> guess that's part of the question, whether those classes count as User API
> or Developer API.
> >
> > I took a quick glance at the operator you mentioned, but I'm hoping I
> can make use of Flink's new support for pod templates to make it as similar
> as possible to a native Deployment resource.
> >
> > Regards,
> > Alexis.
> > ________
> > From: Roman Khachatryan
> > Sent: Friday, July 2, 2021 9:19 PM
> > To: Alexis Sarda-Espinosa ; Yang
> Wang
> > Cc: user@flink.apache.org
> > Subject: Re: Using Flink's Kubernetes API inside Java
> >
> > Hi Alexis,
> >
> > Have you looked at flink-on-k8s-operator [1]?
> > It seems to have the functionality you need:
> >
> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569
> >
> > I couldn't find many Flink-specific classes in the operator you
> > mentioned, but in general classes annotated with Public and
> > PublicEvolving are unlikely to change if that's your concern.
> >
> > Also pulling in Yang Wang.
> >
> > [1]
> > https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/
> >
> >
> > Regards,
> > Roman
> >
> >
> > On Thu, Jul 1, 2021 at 7:49 PM Alexis Sarda-Espinosa
> > wrote:
> > >
> > > Hello everyone,
> > >
> > > I'm testing a custom Kubernetes operator that should fulfill some
> specific requirements I have for Flink. I know of this WIP project:
> https://github.com/wangyang0918/flink-native-k8s-operator
> > >
> > > I can see that it uses some classes that aren't publicly documented,
> and I believe it doesn't cover Flink K8s sessions, which I would like to
> use.
> > >
> > > Is there something I could use for Flink K8s sessions? And is it ok if
> I use these classes knowing that I might need adjustments for future Flink
> versions?
> > >
> > > Regards,
> > > Alexis.
> > >
>
Re: Using Flink's Kubernetes API inside Java
Hi Alexis, KubernetesSessionCli provides a similar functionality IIUC but it's also marked as @Internal (so it likely will change in the future; the REST APIs it uses aren't likely to change, but I guess it doesn't help as you'd like some helper classes.). I think it's a good idea to ask this question on a dev mailing list. Regards, Roman On Fri, Jul 2, 2021 at 11:19 PM Alexis Sarda-Espinosa wrote: > > Hi Roman, > > In the operator I mentioned I see logic like the one here: > https://github.com/wangyang0918/flink-native-k8s-operator/blob/a60a9826d4bcdaa4f23cf296d95954b9f6f328c3/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkApplicationController.java#L169 > > For instance, DefaultClusterClientServiceLoader is annotated with @Internal: > https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/client/deployment/DefaultClusterClientServiceLoader.html > > And just like ApplicationClusterDeployer is in charge of Application Mode > (according to the Javadoc), I was wondering if there's something similar for > Session Mode. > > Perhaps I should subscribe to the developer mailing list? Although I guess > that's part of the question, whether those classes count as User API or > Developer API. > > I took a quick glance at the operator you mentioned, but I'm hoping I can > make use of Flink's new support for pod templates to make it as similar as > possible to a native Deployment resource. > > Regards, > Alexis. > > From: Roman Khachatryan > Sent: Friday, July 2, 2021 9:19 PM > To: Alexis Sarda-Espinosa ; Yang Wang > > Cc: user@flink.apache.org > Subject: Re: Using Flink's Kubernetes API inside Java > > Hi Alexis, > > Have you looked at flink-on-k8s-operator [1]? > It seems to have the functionality you need: > https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569 > > I couldn't find many Flink-specific classes in the operator you > mentioned, but in general classes annotated with Public and > PublicEvolving are unlikely to change if that's your concern. > > Also pulling in Yang Wang. > > [1] > https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/ > > > Regards, > Roman > > > On Thu, Jul 1, 2021 at 7:49 PM Alexis Sarda-Espinosa > wrote: > > > > Hello everyone, > > > > I'm testing a custom Kubernetes operator that should fulfill some specific > > requirements I have for Flink. I know of this WIP project: > > https://github.com/wangyang0918/flink-native-k8s-operator > > > > I can see that it uses some classes that aren't publicly documented, and I > > believe it doesn't cover Flink K8s sessions, which I would like to use. > > > > Is there something I could use for Flink K8s sessions? And is it ok if I > > use these classes knowing that I might need adjustments for future Flink > > versions? > > > > Regards, > > Alexis. > >
Re: Using Flink's Kubernetes API inside Java
Hi Roman, In the operator I mentioned I see logic like the one here: https://github.com/wangyang0918/flink-native-k8s-operator/blob/a60a9826d4bcdaa4f23cf296d95954b9f6f328c3/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkApplicationController.java#L169 For instance, DefaultClusterClientServiceLoader is annotated with @Internal: https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/client/deployment/DefaultClusterClientServiceLoader.html And just like ApplicationClusterDeployer is in charge of Application Mode (according to the Javadoc<https://ci.apache.org/projects/flink/flink-docs-release-1.13/api/java/org/apache/flink/client/deployment/application/cli/ApplicationClusterDeployer.html>), I was wondering if there's something similar for Session Mode. Perhaps I should subscribe to the developer mailing list? Although I guess that's part of the question, whether those classes count as User API or Developer API. I took a quick glance at the operator you mentioned, but I'm hoping I can make use of Flink's new support for pod templates to make it as similar as possible to a native Deployment resource. Regards, Alexis. From: Roman Khachatryan Sent: Friday, July 2, 2021 9:19 PM To: Alexis Sarda-Espinosa ; Yang Wang Cc: user@flink.apache.org Subject: Re: Using Flink's Kubernetes API inside Java Hi Alexis, Have you looked at flink-on-k8s-operator [1]? It seems to have the functionality you need: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569 I couldn't find many Flink-specific classes in the operator you mentioned, but in general classes annotated with Public and PublicEvolving are unlikely to change if that's your concern. Also pulling in Yang Wang. [1] https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/ Regards, Roman On Thu, Jul 1, 2021 at 7:49 PM Alexis Sarda-Espinosa wrote: > > Hello everyone, > > I'm testing a custom Kubernetes operator that should fulfill some specific > requirements I have for Flink. I know of this WIP project: > https://github.com/wangyang0918/flink-native-k8s-operator > > I can see that it uses some classes that aren't publicly documented, and I > believe it doesn't cover Flink K8s sessions, which I would like to use. > > Is there something I could use for Flink K8s sessions? And is it ok if I use > these classes knowing that I might need adjustments for future Flink versions? > > Regards, > Alexis. >
Re: Using Flink's Kubernetes API inside Java
Hi Alexis, Have you looked at flink-on-k8s-operator [1]? It seems to have the functionality you need: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/0310df76d6e2128cd5d2bc51fae4e842d370c463/controllers/flinkcluster_reconciler.go#L569 I couldn't find many Flink-specific classes in the operator you mentioned, but in general classes annotated with Public and PublicEvolving are unlikely to change if that's your concern. Also pulling in Yang Wang. [1] https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/ Regards, Roman On Thu, Jul 1, 2021 at 7:49 PM Alexis Sarda-Espinosa wrote: > > Hello everyone, > > I'm testing a custom Kubernetes operator that should fulfill some specific > requirements I have for Flink. I know of this WIP project: > https://github.com/wangyang0918/flink-native-k8s-operator > > I can see that it uses some classes that aren't publicly documented, and I > believe it doesn't cover Flink K8s sessions, which I would like to use. > > Is there something I could use for Flink K8s sessions? And is it ok if I use > these classes knowing that I might need adjustments for future Flink versions? > > Regards, > Alexis. >
Using Flink's Kubernetes API inside Java
Hello everyone, I'm testing a custom Kubernetes operator that should fulfill some specific requirements I have for Flink. I know of this WIP project: https://github.com/wangyang0918/flink-native-k8s-operator I can see that it uses some classes that aren't publicly documented, and I believe it doesn't cover Flink K8s sessions, which I would like to use. Is there something I could use for Flink K8s sessions? And is it ok if I use these classes knowing that I might need adjustments for future Flink versions? Regards, Alexis.
