ErikBPF commented on issue #7455:
URL:
https://github.com/apache/datafusion-comet/issues/7455#issuecomment-6725614137
I reproduced the Spark 4.2 pairing with both Delta 4.4.0 and 4.4.1 on Comet
1.2.0 source (`ab46419fcc99bcd34aa4d89f7b545d5d51694a88`). Each full contrib
run executes 279 tests: **278 passed, one failed, zero
errors/skips/canceled/ignored/pending**.
The same failure appears in both lanes:
```text
DeltaScanContribSuite:
discovery harness: every credential/encryption-shaped fs.s3a.* key Hadoop's
Constants class declares is either compared by AllS3ConfigKeys or
individually documented as exempt
HashSet("fs.s3a.encryption.context") was not empty
```
NativeScan (118), DML-decline (3) and local MinIO/S3 (3) pass; the 155-case
admission suite contains the failure. Synthetic merge/clustering/hygiene
workloads also pass 37 checks per lane with matching stock outputs, but that
does not override the failed admission audit.
After the matching native/JVM build, the root-reactor selection is:
```bash
set -euo pipefail
./mvnw -Pspark-4.2,delta -Ddelta.artifact=delta-spark_4.2 \
-Ddelta.spark.version=4.4.0 test -Dtest=none \
-DwildcardSuites=org.apache.comet.contrib.delta
# Repeat with -Ddelta.spark.version=4.4.1.
```
Both artifact and version overrides matter. The source-matched packages in
these runs reused the exact released AMD64 JNI; no native rebuild or
current-main runtime is claimed. The actual failed-suite classpath has one
`Constants.class` provider: `hadoop-aws-3.5.0.jar` (SHA256
`66bd9eefb938b1a1657487324b63c13df2d8b17164a1dd24e0ef37b905bc5baa`).
The inspected current-main discovery test and #7026 head also lack
classification for this key. Hadoop sends KMS context on write requests, but
resolves/parses it during initialization. Fifteen pure in-memory Hadoop 3.5
reference checks passed; malformed context raises `IllegalArgumentException`
through `S3AUtils.buildEncryptionSecrets` with unset, AES256, SSE-KMS and
DSSE-KMS algorithms. For example:
```java
Configuration conf = new Configuration(false);
conf.set("fs.s3a.encryption.context", "synthetic-malformed-pair");
S3AUtils.buildEncryptionSecrets("synthetic-bucket", conf); //
IllegalArgumentException
```
Source: [Hadoop encryption
initialization](https://github.com/apache/hadoop/blob/rel/release-3.5.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java),
[context
resolution/parsing](https://github.com/apache/hadoop/blob/rel/release-3.5.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AEncryption.java).
This is pairing-gate evidence, **not** a reproduced Comet read-admission
discrepancy or AWS KMS-policy failure. A blanket exemption cannot be justified
solely because GET/HEAD omit the header. The narrow follow-up is to agree this
property's classification with the S3 gate owner, coordinate with #7026,
preserve existing SSE-C/CSE declines, and rerun the nonempty contrib suites. I
am keeping that follow-up separate from the private-expression feature
requests; no existing assertion was bypassed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
