jackye1995 commented on code in PR #5902: URL: https://github.com/apache/iceberg/pull/5902#discussion_r985267316
########## docs/aws.md: ########## @@ -552,12 +552,13 @@ This also serves as an example for users who would like to implement their own A This client factory has the following configurable catalog properties: -| Property | Default | Description | -| --------------------------------- | ---------------------------------------- | ------------------------------------------------------ | -| client.assume-role.arn | null, requires user input | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume | -| client.assume-role.region | null, requires user input | All AWS clients except the STS client will use the given region instead of the default region chain | -| client.assume-role.external-id | null | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) | -| client.assume-role.timeout-sec | 1 hour | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client. | +| Property | Default | Description | Review Comment: no need to change all the lines, just need to add one more line for the new config added ########## docs/aws.md: ########## @@ -573,6 +574,55 @@ spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:{{% icebergVersio --conf spark.sql.catalog.my_catalog.client.assume-role.region=ap-northeast-1 ``` +### URL Connection HTTP Client +In default, AWS clients use the [URL Connection HTTP Client](https://mvnrepository.com/artifact/software.amazon.awssdk/url-connection-client) for HTTP connection management. + +This HTTP Client has the following configurable properties: + +| Property | Default | Description | +|-------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| http-client.urlconnection.socket-timeout-ms | null | An optional [socket timeout](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.Builder.html#socketTimeout(java.time.Duration)) in milliseconds | +| http-client.urlconnection.connection-timeout-ms | null | An optional [connection timeout](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.Builder.html#connectionTimeout(java.time.Duration)) in milliseconds | + +Here is an example to start Spark shell with URL Connection HTTP client and some optional settings: +```shell +spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:{{% icebergVersion %}},software.amazon.awssdk:bundle:2.17.257 \ + --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \ + --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \ + --conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \ + --conf spark.sql.catalog.my_catalog.http-client.urlconnection.socket-timeout-ms=80 \ + --conf spark.sql.catalog.my_catalog.http-client.urlconnection.connection-timeout-ms=90 +``` + +### Apache HTTP Client Review Comment: I think we want a bit more levels, 1 general level for HTTP client configuration, an intro paragraph about choosing the type of HTTP client, and 2 sub sections for configuring UrlConnection and Apache type clients. ########## docs/aws.md: ########## @@ -552,12 +552,13 @@ This also serves as an example for users who would like to implement their own A This client factory has the following configurable catalog properties: -| Property | Default | Description | -| --------------------------------- | ---------------------------------------- | ------------------------------------------------------ | -| client.assume-role.arn | null, requires user input | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume | -| client.assume-role.region | null, requires user input | All AWS clients except the STS client will use the given region instead of the default region chain | -| client.assume-role.external-id | null | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) | -| client.assume-role.timeout-sec | 1 hour | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client. | +| Property | Default | Description | +|-----------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| client.assume-role.arn | null, requires user input | ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume | +| client.assume-role.region | null, requires user input | All AWS clients except the STS client will use the given region instead of the default region chain | +| client.assume-role.external-id | null | An optional [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) | +| client.assume-role.timeout-sec | 1 hour | Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client. | +| client.assume-role.session-name | null | An optional [session name](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#ck_rolesessionname) | Review Comment: I know you recently added both changes to assume role factory and http client, but they are kind of unrelated. Can we make this only focused on the features in http client? that is already a good amount of changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
