sunchao commented on code in PR #6113:
URL: https://github.com/apache/datafusion-comet/pull/6113#discussion_r4100340045
##########
.github/workflows/iceberg_spark_test_reusable.yml:
##########
@@ -224,7 +227,40 @@ jobs:
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet
cache requires cleanups
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew
-DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }}
-DflinkVersions= -DkafkaVersions= \
:iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short
}}_${{ inputs.scala }}:test \
+ --init-script ../dev/ci/iceberg-test-shards.gradle \
Review Comment:
[P1] Use an extensions-specific partition before enabling this init script.
`CometIcebergTestShard.owner()` reserves shard 1 exclusively for
`TestStructuredStreamingRead*` and assigns everything else to shards 2–4. None
of the configured extensions source sets contains that family. Consequently,
shard 1 becomes `NO-SOURCE`, Gradle skips the `doFirst` inventory writer, and
the new coverage job fails because inventory 1 is missing, even when every
extensions test passes. This makes the Iceberg workflow fail and blocks
merge-queue runs that require it. Preserve the core streaming special case
while distributing extensions across all four shards, and add a fixture without
streaming tests.
Evidence: Checked upstream Iceberg 1.8.1/Spark 3.4, 1.9.1/Spark 3.5,
1.10.0/Spark 3.5, and 1.11.0/Spark 4.1 source trees and Gradle configuration:
`TestStructuredStreamingRead3` belongs to the core module, not extensions.
Reproduced with Gradle 8.14.4/JDK 17 using isolated Java test projects
containing passing `TestExtendedParser`, `TestAddFilesProcedure`, and
`TestAlterTableSchema` tests and the unchanged repository init script with
`-PcometShardTask=:test -PcometShardIndex=1..4 -PcometShardCount=4`. Baseline
executed three tests. Shard 1 reported `:test NO-SOURCE` and wrote no manifest.
Shards 2–4 each passed one test and wrote their inventories. The repository
coverage checker exited 1 with `Expected shard indices [1, 2, 3, 4], found [2,
3, 4]`. Reproduction and logs:
`/tmp/comet-6113-repro-n141sa7s/run-isolated-repro.py` and its `isolated/`
directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
