Re: how to initialize few things at task managers

[huweihua]

yarn.ship-files only works in yarn environment. 

Maybe you could use a custom Docker entry point[1] like Austin said.

[1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#further-customization



> 2022年4月20日 上午1:19，Great Info  写道：
> 
> I am deploying as a docker on our servers, due to some restrictions I can 
> only pass Keystore URLs.  
> 
> one option is yarn.ship-files !.  can you help me with pointing to the sample 
> code, and how job manager can ship this file?
> 
> download as part of job's main function and send to all task managers.. will 
> this work? 

Re: how to initialize few things at task managers

[Great Info]

I am deploying as a docker on our servers, due to some restrictions I can
only pass Keystore URLs.

one option is yarn.ship-files !.  can you help me with pointing to the
sample code, and how job manager can ship this file?

download as part of job's main function and send to all task managers..
will this work?



On Mon, Apr 18, 2022 at 10:10 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> If you are using Kubernetes to deploy Flink, you could think about an
> initContainer on the TMs or a custom Docker entry point that does this
> initialization.
>
> Best,
> Austin
>
> On Mon, Apr 18, 2022 at 7:49 AM huweihua  wrote:
>
>> Hi, Init stuff when task manager comes up is not an option.
>> But if the Keystore file is not changeable and you are using yarn mode,
>> maybe you can use ‘yarn.ship-files’[1] to localize it.
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/config/#yarn-ship-files
>>
>> 2022年4月16日 下午11:44，Great Info  写道：
>>
>> I need to download Keystore and use it while creating the source
>> connector, currently, I am overriding the open method
>> 
>>  but
>> this gets called for each of my source connectors.
>>
>>  @Override
>> public void open(Configuration configuration) throws Exception {
>>
>>   // do few things like download Keystore to default path etc
>>  super.open(configuration)
>> }
>>
>> Is there an option to init a few pre stuff as soon as the task manager
>> comes up?
>>
>>
>>

Re: how to initialize few things at task managers

[Austin Cawley-Edwards]

If you are using Kubernetes to deploy Flink, you could think about an
initContainer on the TMs or a custom Docker entry point that does this
initialization.

Best,
Austin

On Mon, Apr 18, 2022 at 7:49 AM huweihua  wrote:

> Hi, Init stuff when task manager comes up is not an option.
> But if the Keystore file is not changeable and you are using yarn mode,
> maybe you can use ‘yarn.ship-files’[1] to localize it.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/config/#yarn-ship-files
>
> 2022年4月16日 下午11:44，Great Info  写道：
>
> I need to download Keystore and use it while creating the source
> connector, currently, I am overriding the open method
> 
>  but
> this gets called for each of my source connectors.
>
>  @Override
> public void open(Configuration configuration) throws Exception {
>
>   // do few things like download Keystore to default path etc
>  super.open(configuration)
> }
>
> Is there an option to init a few pre stuff as soon as the task manager
> comes up?
>
>
>

Re: how to initialize few things at task managers

[huweihua]

Hi, Init stuff when task manager comes up is not an option.
But if the Keystore file is not changeable and you are using yarn mode, maybe 
you can use ‘yarn.ship-files’[1] to localize it.

[1]https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/config/#yarn-ship-files

> 2022年4月16日 下午11:44，Great Info  写道：
> 
> I need to download Keystore and use it while creating the source connector, 
> currently, I am overriding the open method 
> 
>  but this gets called for each of my source connectors.
> 
>  @Override
> public void open(Configuration configuration) throws Exception {
> 
>   // do few things like download Keystore to default path etc 
>  super.open(configuration)
> }
> Is there an option to init a few pre stuff as soon as the task manager comes 
> up?