yikf commented on issue #6970: URL: https://github.com/apache/incubator-gluten/issues/6970#issuecomment-2511452471
Hi all, we will continue with this work, and we plan to discuss the following points. ## Project engineering Split the Gluten config and from a module perspective, we assigned the gluten, velox, and ch-related configurations to their respective modules. At the level of a single module, we classified the configurations into two classes, static and non-static. ## Configuration Specifications Standardize configuration naming, and prefix all configuration entries with `spark.gluten`. As described in https://github.com/apache/incubator-gluten/issues/6970#issuecomment-2304090991. Now we are leaning towards Option 2, and if there are no other objections, we will standardize based on Option 2. If there are other ideas, we can initiate an email vote. Option 1 1. _**common config:**_ spark.gluten.sql.columnar.xxx => spark.gluten.xxx 2. **_backend config:_** spark.gluten.sql.columnar.backend.velox.xxx => spark.gluten.backend.velox.xxx Option 2 1. _**common config:**_ spark.gluten.sql.columnar.xxx => spark.gluten.common.xxx 2. **_backend config:_** spark.gluten.sql.columnar.backend.velox.xxx => spark.gluten.velox..xxx ## Backward compatibility When configuring key modifications, we need to be compatible with the old keys and output deprecation information, gradually phasing them out in subsequent versions. ## Summary & Implementation To achieve greater configuration flexibility, I initially wanted to reimplement ConfigEntry. The advantage of this approach is that we can customize attributes, such as "backend" to identify more customized information. However, Gluten's configuration comes from Spark, which makes it seem unlikely that we can reimplement ConfigEntry. For example, we cannot identify static configuration if we do so. If we directly base our ConfigEntry on Spark, it may not be flexible enough, such as not being able to record expiration information. Any thoughs? @zhztheplayer @Yohahaha @zml1206 @jackylee-ch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
