Re: [DISCUSS] [statefun] resolve placeholders in module.yaml
Hi Igal
I created a jira and already started looking into what it would take to
implement it. Please kindly assign it to me when you get the chance
https://issues.apache.org/jira/browse/FLINK-26570
Thanks!
Fil
On Wed, 9 Mar 2022 at 15:08, Igal Shilman wrote:
> Hello Fil,
> I think that adding a very simple interpolation mechanism for remote
> functions can be useful.
> And also your suggested place should be good.
> Can you create a JIRA issue with this description, and there we can
> continue the conversation and scope this?
>
> Thanks!
> Igal.
>
>
> On Mon, Mar 7, 2022 at 5:50 PM Filip Karnicki
> wrote:
>
> > Hi, as far as I can tell, the way to provide a keystore/truststore
> password
> > to the kafka ingress/egress config is to put it in plaintext in
> > module.yaml, like so:
> >
> > kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
> > ssl.truststore.password: changeme
> >
> > This isn't ideal and I think it would be neater to be able to replace a
> > placeholder with something from the parameter tool / global config
> >
> > kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
> > ssl.truststore.password: ${SSL_TRUSTSTORE_PASS}
> >
> > Similarly, we need to get our hands on a kerberos keytab location inside
> > module.yaml. This is not a problem when the location is static and
> > available to all cluster nodes, but when yarn gets involved, it's only
> the
> > yarn client (?) that has the keytab file in a static location. As far as
> I
> > can tell, task manager nodes get a 'resolved' and node/container-specific
> > location, something along the lines of
> > "/JBOD_D01/yarn/application_12345667_0001", which is different for every
> > node. I think I could get my hands on that location from the global
> config,
> > seeing as YarnTaskExecutorRunner sets
> > '-Dsecurity.kerberos.login.keytab=/container/specific/path/here'
> >
> > To achieve all of this, we could alter RemoteModule#bindComponent to
> > replace instances of ${PLACEHOLDERs} with values from the global config
> > using regex.
> >
> > Please let me know what you think
> > Fil
> >
>
Re: [DISCUSS] [statefun] resolve placeholders in module.yaml
Hello Fil,
I think that adding a very simple interpolation mechanism for remote
functions can be useful.
And also your suggested place should be good.
Can you create a JIRA issue with this description, and there we can
continue the conversation and scope this?
Thanks!
Igal.
On Mon, Mar 7, 2022 at 5:50 PM Filip Karnicki
wrote:
> Hi, as far as I can tell, the way to provide a keystore/truststore password
> to the kafka ingress/egress config is to put it in plaintext in
> module.yaml, like so:
>
> kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
> ssl.truststore.password: changeme
>
> This isn't ideal and I think it would be neater to be able to replace a
> placeholder with something from the parameter tool / global config
>
> kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
> ssl.truststore.password: ${SSL_TRUSTSTORE_PASS}
>
> Similarly, we need to get our hands on a kerberos keytab location inside
> module.yaml. This is not a problem when the location is static and
> available to all cluster nodes, but when yarn gets involved, it's only the
> yarn client (?) that has the keytab file in a static location. As far as I
> can tell, task manager nodes get a 'resolved' and node/container-specific
> location, something along the lines of
> "/JBOD_D01/yarn/application_12345667_0001", which is different for every
> node. I think I could get my hands on that location from the global config,
> seeing as YarnTaskExecutorRunner sets
> '-Dsecurity.kerberos.login.keytab=/container/specific/path/here'
>
> To achieve all of this, we could alter RemoteModule#bindComponent to
> replace instances of ${PLACEHOLDERs} with values from the global config
> using regex.
>
> Please let me know what you think
> Fil
>
[DISCUSS] [statefun] resolve placeholders in module.yaml
Hi, as far as I can tell, the way to provide a keystore/truststore password
to the kafka ingress/egress config is to put it in plaintext in
module.yaml, like so:
kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
ssl.truststore.password: changeme
This isn't ideal and I think it would be neater to be able to replace a
placeholder with something from the parameter tool / global config
kind: io.statefun.kafka.v1/ingressspec: #(...) properties:-
ssl.truststore.password: ${SSL_TRUSTSTORE_PASS}
Similarly, we need to get our hands on a kerberos keytab location inside
module.yaml. This is not a problem when the location is static and
available to all cluster nodes, but when yarn gets involved, it's only the
yarn client (?) that has the keytab file in a static location. As far as I
can tell, task manager nodes get a 'resolved' and node/container-specific
location, something along the lines of
"/JBOD_D01/yarn/application_12345667_0001", which is different for every
node. I think I could get my hands on that location from the global config,
seeing as YarnTaskExecutorRunner sets
'-Dsecurity.kerberos.login.keytab=/container/specific/path/here'
To achieve all of this, we could alter RemoteModule#bindComponent to
replace instances of ${PLACEHOLDERs} with values from the global config
using regex.
Please let me know what you think
Fil
