JonasJ-ap opened a new pull request, #6746:
URL: https://github.com/apache/iceberg/pull/6746
## Problem Addressed
This PR fix the problem described in issue #6715 by using reflection to
instantiate the httpclient configuration impl class to avoid runtime deps of
both `url-connection-client` and `apache-httpclient`
## Test Environment
Spark3.3, Scala 2.12.
scripts to spawn spark shell:
```bash
BRANCH_NAME=bug_fix_aws_httpclient_conflict_prefix_map
DEPENDENCIES=""
# add AWS dependnecy
AWS_SDK_VERSION=2.17.257
AWS_MAVEN_GROUP=software.amazon.awssdk
AWS_PACKAGES=(
"apache-client"
"s3"
"glue"
"kms"
"iam"
"sts"
"dynamodb"
)
for pkg in "${AWS_PACKAGES[@]}"; do
DEPENDENCIES+="$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION,"
done
JARS="iceberg-spark-runtime-3.3_$BRANCH_NAME.jar"
# start Spark SQL client shell
spark-shell --packages=$DEPENDENCIES --jars=$JARS\
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
--conf
spark.sql.catalog.demo.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
--conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.aws.s3.S3FileIO
\
--conf spark.sql.catalog.demo.warehouse=s3://gluetestjonas/warehouse \
--conf spark.sql.catalog.demo.http-client.type=apache
```
### Test commands:
```bash
scala> val data = spark.range(0, 10)
scala> data.writeTo("demo.default.test1").create()
```
### Before the fix:
will raise `java.lang.NoClassDefFoundError`
```bash
java.lang.NoClassDefFoundError:
software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient$Builder
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at
java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3458)
at java.base/java.lang.Class.getDeclaredMethod(Class.java:2726)
at
java.base/java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1525)
at
java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:413)
at
java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:384)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at
java.base/java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:384)
at
java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:110)
at
java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:107)
at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:73)
at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:70)
at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:229)
at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:211)
at java.base/java.lang.ClassValue.get(ClassValue.java:117)
at java.base/java.io.ClassCache.get(ClassCache.java:84)
at
java.base/java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:363)
at
```
### After the fix
Successfully create the table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
