Hi all, Thanks for looking into this. Yeah, I kept trying different variations of the replacement fields with no success. I'm trying to use the .getenv() technique now but our cluster is having problems and I haven't been able to reinstall the operator. I'll reply once it's all working.
Thanks, Francis On Thu, 5 May 2022 at 03:23, Meissner, Dylan <dylan.t.meiss...@nordstrom.com> wrote: > Flink deployment resources support env interpolation natively using $() > syntax. I expected this to "just work" like other resources when using the > operator, but it does not. > > > https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments > > job: > jarURI: local:///my.jar > entryClass: my.JobMainKt > args: > - "--kafka.bootstrap.servers" > - "my.kafka.host:9093" > - "--kafka.sasl.username" > - "$(KAFKA_SASL_USERNAME)" > - "--kafka.sasl.password" > - "$(KAFKA_SASL_PASSWORD)" > > > It would be a great addition, simplifying job startup decision-making > while following existing conventions. > > ------------------------------ > *From:* Yang Wang <danrtsey...@gmail.com> > *Sent:* Tuesday, May 3, 2022 7:22 AM > *To:* Őrhidi Mátyás <matyas.orh...@gmail.com> > *Cc:* Francis Conroy <francis.con...@switchdin.com>; user < > user@flink.apache.org> > *Subject:* Re: Using the official flink operator and kubernetes secrets > > Flink could not support environment replacement in the args. I think you > could access the env via "*System.getenv()*" in the user main method. > It should work since the user main method is executed in the JobManager > side. > > Best, > Yang > > Őrhidi Mátyás <matyas.orh...@gmail.com> 于2022年4月28日周四 19:27写道： > > Also, > > just declaring it in the flink configs should be sufficient, no need to > define it in the pod templates: > > flinkConfiguration: > kubernetes.env.secretKeyRef: > "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" > > > Best, > Matyas > > On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <matyas.orh...@gmail.com> > wrote: > > Hi Francis, > > I suggest accessing the environment variables directly, no need to pass > them as command arguments I guess. > > Best, > Matyas > > On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy < > francis.con...@switchdin.com> wrote: > > Hi all, > > I'm trying to use a kubernetes secret as a command line argument in my job > and the text replacement doesn't seem to be happening. I've verified > passing the custom args via the command line on my local flink cluster but > can't seem to get the environment var replacement to work. > > apiVersion: flink.apache.org/v1alpha1 > kind: FlinkDeployment > metadata: > namespace: default > name: http-over-mqtt > spec: > image: flink:1.14.4-scala_2.12-java11 > flinkVersion: v1_14 > flinkConfiguration: > taskmanager.numberOfTaskSlots: "2" > kubernetes.env.secretKeyRef: > "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" > # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN" > serviceAccount: flink > jobManager: > replicas: 1 > resource: > memory: "1024m" > cpu: 1 > taskManager: > resource: > memory: "1024m" > cpu: 1 > podTemplate: > spec: > serviceAccount: flink > containers: > - name: flink-main-container > volumeMounts: > - mountPath: /flink-job > name: flink-jobs > env: > - name: DJANGO_TOKEN # kubectl create secret generic > switchdin-django-token --from-literal=token='[TOKEN]' > valueFrom: > secretKeyRef: > name: switchdin-django-token > key: token > optional: false > initContainers: > - name: grab-mqtt-over-http-jar > image: docker-push.k8s.local/test/switchdin/platform_flink:job-41 > command: [ "/bin/sh", "-c", > "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in > the init container to the flink-jobs volume > volumeMounts: > - name: flink-jobs > mountPath: /tmp/job > volumes: > - name: flink-jobs > emptyDir: { } > job: > jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar > args: ["--swit-django-token", "$DJANGO_TOKEN", > "--swit-prod","false"] > entryClass: org.switchdin.HTTPOverMQTT > parallelism: 1 > upgradeMode: stateless > state: running > > In the logs I can see: > > 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {} > 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - > --swit-django-token > 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN > 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod > 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false > > Anyone know how I can do this? I'm considering mounting it in a volume, > but that seems like a lot of hassle for such a small thing. > > Thanks in advance! > > > This email and any attachments are proprietary and confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily reflect or represent those of SwitchDin Pty Ltd. If you have > received this email in error, please let us know immediately by reply email > and delete it from your system. You may not use, disseminate, distribute or > copy this message nor disclose its contents to anyone. > SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 > Australia > > -- This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
