dnskr opened a new issue, #6123: URL: https://github.com/apache/kyuubi/issues/6123
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### What would you like to be improved? There is no clear common approach at the moment on how to configure Kyuubi deployment if the Helm chart is used. I would like to discuss requirements, limitations and different options to choose one approach to follow and support it in the Kyuubi Helm chart configuration. The problem has been mentioned and discussed in multiple issues and PRs, so the idea is to collect all opinions in one place and make the decision. **Configuration system of the Apache Kyuubi** The configuration system allows to configure values using the following options (ordered from low to high prio): 1. [static] Kyuubi configuration files 2. [static] Hadoop configuration files 3. [static] Engine (Spark, Flink, Trino etc) configuration files 4. [runtime] Environment variables 5. [runtime] JDBC Connection URLs 6. [runtime] SET Statements **Runtime options `JDBC Connection URLs` and `SET Statements`** Can be skipped in the discussion, because they used only when Kyuubi is up and running. **Runtime option `Environment variables`** Configured by `{{ .Values.env }}` and `{{ .Values.envFrom }}` value properties. The Helm chart users can specify environment variables to provide necessary configuration values with low effort if needed. The properties also allow to use provided (existing) `ConfigMaps` and `Secrets` as the sources of environment variables, for instance: ```yaml env: - name: ENV_VALUE value: env-value - name: ENV_FROM_CONFIGMAP_KEY valueFrom: configMapKeyRef: name: env-configmap key: env-key envFrom: - configMapRef: name: all-env-configmap ``` **Static options** Represented by configuration files which should be located in each Kyuubi container in specific paths. In general case, the easiest way to provide files into Kubernetes pod (container) is to mount `ConfigMap` or `Secret` to a specific path. **[IN PROGRESS] Requirements** _Note: this section is in progress and subject to discuss._ 1. Ability to create `ConfigMaps` under the hood from value properties of `value.yaml` file. `Secrets` should never be created and managed by Helm chart because of security consideration! 2. Ability to specify existing (created outside the chart) `ConfigMaps` and `Secrets` by resource name as a reference. 3. Ability to provide multiple existing `ConfigMaps` and `Secrets` with priority order. Multiple `ConfigMaps` and `Secrets` might have key duplicates, so the implementation should clearly resolve the collision by merging keys in priority order. 4. Ability to mix `ConfigMaps` managed by the chart with `ConfigMaps` and `Secrets` provided by user with priority order. The issue with key duplicates should be clearly resolved. `ConfigMaps` managed by the chart should have the lowest prio. 5. The approach should work for Helm and GitOps tools like ArgoCD, Flux etc. 6. Easy way to specify one or many configuration files as Helm values, i.e. properties in `value.yaml` file. Some configuration files might be huge and complex, so the idea is to prevent identation issues in `values.yaml` file. 7. Easy way to create `ConfigMaps` and `Secrets` from one or many configuration files. Users might have a lot of xml, properties and other files, so the idea is to help users to create `ConfigMap` and `Secret` resources in a simple way. ### How should we improve? **[IN PROGRESS] Approach** _Note: this section is in progress and subject to discuss._ 1. Group configuration file properties in `values.yaml` by system like Kyuubi, Hadoop, Spark, Trino etc. ```yaml kyuubiConfDir: /opt/kyuubi/conf kyuubiConf: ... sparkConfDir: /opt/spark/conf sparkConf: ... ``` 2. Use `files` property to specify various files. Users can define files with any file name. Each entity within `files` property used as a key/value pair in the corresponding `ConfigMap`. ```yaml sparkConf: files: 'spark-env.sh': | #!/usr/bin/env bash export SPARK_LOG_DIR=/opt/spark/logs 'spark-defaults.conf': | spark.submit.deployMode=cluster ``` 3. Use `from` property to specify list of existing `ConfigMaps` and `Secrets` to be mounted to the configuration path of Kyuubi container. ```yaml sparkConf: from: - configMap: name: my-spark-confs - secret: name: my-sensetive-spark-confs - secret: name: my-sensetive-spark-confs-2 items: - key: secretKey path: filename.xml ``` The implementation idea is to use [Projected Volumes](https://kubernetes.io/docs/concepts/storage/projected-volumes/) with `core/v1/SecretProjection` and `core/v1/ConfigMapProjection` entities. Also it will allow to merge `ConfigMap` created from `files` property with the entities from `from` property. 4. Move `xxxConfDir` property to `xxxConf` property. ```yaml sparkConf: dir: /opt/spark/conf ``` 5. Configuration example for Spark ```yaml sparkConf: dir: /opt/spark/conf files: 'spark-env.sh': | #!/usr/bin/env bash export SPARK_LOG_DIR=/opt/spark/logs 'spark-defaults.conf': | spark.submit.deployMode=cluster from: - configMap: name: my-spark-confs - secret: name: my-sensetive-spark-confs - secret: name: my-sensetive-spark-confs-2 items: - key: secretKey path: filename.xml ``` 6. Provide documentation with examples on how to set file content as a property when installing the chart, see [ Helm docs](https://helm.sh/docs/helm/helm_install/#:~:text=can%20use%20%27--set-file%27%20to%20set%20individual%20values%20from%20a%20file). ```sh helm install kyuubi charts/kyuubi --set-file kyuubiConf.log4j2=kyuubi/conf/log4j2.xml.template ``` 7. Provide documentation with examples on how to create `ConfigMap` from file or directory, see [Kubernetes docs ](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#create-configmaps-from-directories). ```sh kubectl create configmap my-spark-configs --from-file=prod/spark-configs/ ``` ### Are you willing to submit PR? - [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
