sunchao commented on code in PR #5557:
URL: https://github.com/apache/datafusion-comet/pull/5557#discussion_r3890175134
##########
spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py:
##########
@@ -84,6 +86,10 @@ def _build_spark() -> SparkSession:
.config("spark.plugins", "org.apache.spark.CometPlugin")
.config("spark.comet.enabled", "true")
.config("spark.comet.exec.enabled", "true")
+ .config(
+ "spark.sql.execution.arrow.useLargeVarTypes",
+ os.environ.get("BENCHMARK_LARGE_VAR_TYPES", "false"),
+ )
Review Comment:
Moved with the dictionary fix to #5560. That PR adds a `dictionary JVM
shuffle` workload with low-cardinality strings, forces the JVM shuffle and
dictionary ratio, requires `CometColumnarExchange`, and supports selecting just
that workload.
On my local Spark 4.0.4 `local[2]` debug build with 2,000,000 rows, two
warmups, and five measured iterations, median end-to-end time was 2.453 s
vanilla / 1.794 s accelerated (1.37x) for `mapInArrow`, and 2.415 s / 1.797 s
(1.34x) for `mapInPandas`. I put the exact scope and numbers in #5560's
description rather than carrying dictionary performance material in this
feature PR.
##########
spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala:
##########
@@ -265,6 +285,317 @@ class CometArrowPythonRunnerSuite extends AnyFunSuite
with Matchers {
}
}
+ for {
+ failSerialization <- Seq(false, true)
+ useLargeVarTypes <- Seq(false, true)
+ } {
+ test(
+ "dictionary inputs materialize logical values " +
Review Comment:
Both guards now live with the code they cover. #5560 owns a real JVM-shuffle
regression module for `mapInArrow` and `mapInPandas`; the workflow runs it in a
separate process for each Spark 4.0/4.1 worker so it gets the Comet shuffle
manager instead of inheriting the main module's shuffle-disabled session.
This feature PR now has the mixed-layout JVM test on its narrow head. It
runs regular and large modes through `struct<string,int,binary>`,
`list<string>`, `map<string,string>`, `list<struct<string>>`, `NullVector`,
`FixedSizeBinaryVector`, and a trailing string, with exact values/types,
recursive source refcounts, and writer-allocation cleanup. The focused feature
suites pass 21/21.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
