andygrove commented on code in PR #7123:
URL: https://github.com/apache/datafusion-comet/pull/7123#discussion_r4357196914
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -1269,16 +1281,34 @@ object CometConf extends ShimCometConf {
conf("spark.comet.convert.broadcastBuildSide.enabled")
.category(CATEGORY_EXEC)
.doc("When enabled, Comet converts a file or V2 scan that it cannot scan
natively (for " +
- "example a Text source) to Arrow when that scan feeds a broadcast
join's build side and " +
+ "example an ORC source) to Arrow when that scan feeds a broadcast
join's build side and " +
"every scan on the probe side is already native, so the broadcast join
runs natively " +
"over the large probe. The build side is usually small
(auto-broadcasts are capped by " +
"the broadcast threshold), so the row to Arrow copy is usually cheap -
though an " +
"explicit BROADCAST hint or a selective filter above the scan can make
it larger. With " +
"dynamic partition pruning the dimension is also read a second time
for the pruning " +
- "subquery. Set this to false to keep such build sides on Spark.")
+ "subquery. Parquet, JSON and CSV scans are converted only by their own
" +
+ "`spark.comet.convert` configs. A Text scan is converted only when " +
+ "`spark.comet.convert.broadcastBuildSide.text.enabled` is also
enabled, because Text " +
+ "files can hold bytes that are not valid UTF-8. Set this to false to
keep such build " +
+ "sides on Spark.")
.booleanConf
.createWithDefault(true)
+ val COMET_CONVERT_BROADCAST_BUILD_SIDE_TEXT_ENABLED: ConfigEntry[Boolean] =
Review Comment:
Yes, this makes sense. I think the extra config is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
