pedrumj2 commented on code in PR #13613:
URL: https://github.com/apache/gluten/pull/13613#discussion_r4570110994
##########
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -613,6 +612,21 @@ object GlutenConfig extends ConfigRegistry {
nativeConfMap
.put(SQLConf.LEGACY_TIME_PARSER_POLICY.key,
v.toUpperCase(Locale.ROOT)))
+ // Spark accepts this value in any case, and native code compares it in
upper case.
+ nativeConfMap.put(
+ SQLConf.MAP_KEY_DEDUP_POLICY.key,
+ conf
+ .getOrElse(
+ SQLConf.MAP_KEY_DEDUP_POLICY.key,
+ SQLConf.MAP_KEY_DEDUP_POLICY.defaultValueString)
+ .toUpperCase(Locale.ROOT)
+ )
+
+ // Spark accepts the codec name in any case, and Velox matches it in lower
case.
+ conf
Review Comment:
Thank for the review @rui-mo
> Would it make sense to make name matching case-insensitive in Velox or
elsewhere, so we can avoid having to use toUpperCase in some places and
toLowerCase in others?
Thats a good point, there are a few different options but please let me know
if you feel this is not the right repository for this change:
### Option1 Handle in the Engines
- Every engine would need to handle the case translation on their end
(Velox, Bolt, ...)
- The `spark.io.compression.codec` can be handled in the engines since we
pass down the value as is but the `spark.sql.mapKeyDedupPolicy` might be a bit
more involved since [we convert it into a
"true/false"](https://github.com/apache/gluten/blob/3dae1409f9c8656dbb79d26f31ad5b4b8e0f3dbc/cpp/velox/compute/WholeStageResultIterator.cc#L692)
value in gluten
### Option2 Handle in Gluten
This was the path in this PR
### Option3: Spark
Leave it to Spark to ensure all spark configs are handle in a consistent
way. Today it seems some configs are changed to uppercase while others are not.
For example
[mapKeyDedupPolicy](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3785-L3794)
```
val MAP_KEY_DEDUP_POLICY = buildConf("spark.sql.mapKeyDedupPolicy")
.doc("The policy to deduplicate map keys in builtin function: CreateMap,
MapFromArrays, " +
"MapFromEntries, StringToMap, MapConcat and TransformKeys. When
EXCEPTION, the query " +
"fails if duplicated map keys are detected. When LAST_WIN, the map key
that is inserted " +
"at last takes precedence.")
.version("3.0.0")
.stringConf
transform(_.toUpperCase(Locale.ROOT))
.checkValues(MapKeyDedupPolicy.values.map(_.toString))
.createWithDefault(MapKeyDedupPolicy.EXCEPTION.toString)
```
But this route would only work for Spark versions going forward and we'd
still need some way to handle the previous versions.
Please let me know what you think, thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]