[
https://issues.apache.org/jira/browse/FLINK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524189#comment-17524189
]
Biao Geng commented on FLINK-27303:
-----------------------------------
Hi [~gyfora], big +1 for this jira. And I believe besides log config files,
there will also be a lot of pod template file if we define it the yaml. Because
the {{applyCommonPodTemplate}} will create temporary files for pod template as
well.
I think your suggestion makes sense. And one possible solution is to divide the
generated configs into 2 types: type 1 is configs needed by observer and
reconciler. type 2 is configs that not used by observer or reconciler.
Besides, I am wondering if it is a good idea to maintain a concurrent hashmap
to save the effective config of each deployment so that we can reduce some
repeated creation of effective configs. It may be the bottleneck of memory
usage when there are plenty of deployments in the operator and make the
operator less `stateless`. Not sure if it is worthwhile.
> Flink Operator will create a large amount of temp log config files
> ------------------------------------------------------------------
>
> Key: FLINK-27303
> URL: https://issues.apache.org/jira/browse/FLINK-27303
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.0.0
> Reporter: Gyula Fora
> Priority: Critical
> Fix For: kubernetes-operator-1.0.0
>
>
> Now we use the configbuilder in multiple different places to generate the
> effective config including observer, reconciler, validator etc.
> The effective config gerenration logic also creates temporary log config
> files (if spec logConfiguration is set) which would lead to 3-4 files
> generated in every reconcile loop for a given job. These files are not
> cleaned up until the operator restarts leading to a large amount of files.
> I believe we should change the config generation logic and only apply the
> logconfig generation logic right before flink cluster submission as that is
> the only thing affected by it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
