Re: Encryption of parameters in flink-conf.yaml

Hi, Thanks for the reply. I don't think I can use IAM integration and avoid distributing keys to the application because my Flink application is running outside AWS EC2, in native K8s cluster nodes, from where I am distributing to S3 services hosted on AWS. If there is a procedure to still integr

Re: Encryption of parameters in flink-conf.yaml

hi Anuj, As Martijn said IAM is the preferred option but if you've no other way than access keys then environment variables is a better choice. Such case conf doesn't contain plain text keys. Just a side note, putting `s3a.access.key` into Flink conf file is not configuring Hadoop S3. The way how

Re: Encryption of parameters in flink-conf.yaml

Hi Anuj, You can't provide the values for S3 in job code, since the S3 filesystems are loaded via plugins. Credentials must be stored in flink-conf.yaml. The recommended method for setting up credentials is by using IAM, not via Access Keys. See https://nightlies.apache.org/flink/flink-docs-master

Re: Encryption of parameters in flink-conf.yaml

Hi, Thanks for the reply. Yes my flink deployment is on K8s but I am not using Flink-k8s operator. If i understood correctly, even with init-container the flink-conf.yaml (inside the container) would finally contain unencrypted values for access tokens. We don't want to persist such sensitive dat

Re: Encryption of parameters in flink-conf.yaml

Hi Anuj, To my best knowledge, flink does not provide the encryption strategy support for now. If you are using flink on k8s, it is possible to achieve the encryption of parameters using the init container. You can check this SO

Encryption of parameters in flink-conf.yaml

Hi Community, I am trying to create an amazon S3 filesystem distributor using flink and for this I am using hadoop S3a connector with Flink filesystem sink. My flink application would run in a non-AWS environment, on native cluster; so I need to put my access keys in flink configuration. For conne