[GitHub] [iceberg] JonasJ-ap opened a new pull request, #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

[via GitHub]

JonasJ-ap opened a new pull request, #6746:
URL: https://github.com/apache/iceberg/pull/6746

   ## Problem Addressed
   This PR fix the problem described in issue #6715 by using reflection to 
instantiate the httpclient configuration impl class to avoid runtime deps of 
both `url-connection-client` and `apache-httpclient`
   
   ## Test Environment
   Spark3.3, Scala 2.12.
   
   scripts to spawn spark shell:
   ```bash
   BRANCH_NAME=bug_fix_aws_httpclient_conflict_prefix_map
   DEPENDENCIES=""
   
   # add AWS dependnecy
   AWS_SDK_VERSION=2.17.257
   AWS_MAVEN_GROUP=software.amazon.awssdk
   AWS_PACKAGES=(
       "apache-client"
       "s3"
       "glue"
       "kms"
       "iam"
       "sts"
       "dynamodb"
   )
   for pkg in "${AWS_PACKAGES[@]}"; do
       DEPENDENCIES+="$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION,"
   done
   
   JARS="iceberg-spark-runtime-3.3_$BRANCH_NAME.jar"
   
   # start Spark SQL client shell
   spark-shell --packages=$DEPENDENCIES --jars=$JARS\
       --conf 
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
 \
       --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
       --conf 
spark.sql.catalog.demo.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
       --conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.aws.s3.S3FileIO 
\
       --conf spark.sql.catalog.demo.warehouse=s3://gluetestjonas/warehouse \
       --conf spark.sql.catalog.demo.http-client.type=apache
   ```
   ### Test commands:
   ```bash
   scala> val data = spark.range(0, 10)
   scala> data.writeTo("demo.default.test1").create()
   ```
   ### Before the fix:
   will raise `java.lang.NoClassDefFoundError`
   ```bash
   java.lang.NoClassDefFoundError: 
software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient$Builder
           at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
           at 
java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3458)
           at java.base/java.lang.Class.getDeclaredMethod(Class.java:2726)
           at 
java.base/java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1525)
           at 
java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:413)
           at 
java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:384)
           at 
java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
           at 
java.base/java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:384)
           at 
java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:110)
           at 
java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:107)
           at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:73)
           at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:70)
           at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:229)
           at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:211)
           at java.base/java.lang.ClassValue.get(ClassValue.java:117)
           at java.base/java.io.ClassCache.get(ClassCache.java:84)
           at 
java.base/java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:363)
           at 
   ```
   ### After the fix
   Successfully create the table
   ![Screen Shot 2023-02-04 at 18 43 
55](https://user-images.githubusercontent.com/55990951/216794213-9a88ebd6-90da-4166-89d0-5696049e8c94.png)
   ![Screen Shot 2023-02-04 at 18 44 
35](https://user-images.githubusercontent.com/55990951/216794249-f1f3c1e1-1c2a-428a-9102-cbd676b14aae.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org