andygrove commented on PR #7336:
URL: 
https://github.com/apache/datafusion-comet/pull/7336#issuecomment-6922289932

   Thanks. I merged main to resolve the conflict. The only conflict was in the 
1.3.0 section of the upgrade guide, where I kept main's Spark 3.4 removal note 
and deprecated-settings entry next to the shuffle manager entry. Since 3.4 is 
gone, I also updated the FAQ to say only Spark 3.5 needs the application to set 
the manager. The push started the fork run: 
https://github.com/andygrove/datafusion-comet/actions/runs/43903474356
   
   I also ran it on a Spark 4.0.4 standalone cluster (one master and one 
worker, with two executors, all in separate JVMs on one machine). It used the 
plugin, `spark.plugins=org.apache.spark.CometPlugin`, with no 
`spark.shuffle.manager` set. I ran it once with `--jars` plus `extraClassPath` 
and once with `--jars` only, and both gave the same result:
   
   - `sc.getConf.get("spark.shuffle.manager")` on the driver: 
`CometShuffleManager`
   - `sc.parallelize(1 to 100, 10).map(_ => 
SparkEnv.get.shuffleManager.getClass.getName).distinct.collect` returns only 
`org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager`. Executors 0 
and 1 both report it.
   - `SELECT k, count(*), sum(v) FROM parquet GROUP BY k ORDER BY k` over 1M 
rows gives correct results. The plan is `CometNativeScan` → partial 
`CometHashAggregate` → `CometExchange hashpartitioning … CometNativeShuffle` → 
final `CometHashAggregate` → `CometExchange rangepartitioning … 
CometNativeShuffle` → `CometSort`.
   
   The executor logs show `CometExecutorPlugin` initializing and the native 
library loading, with no errors. This was all on one machine, not a multi-host 
cluster, but executors did not share the driver's `SparkEnv`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to