Copilot commented on code in PR #11423:
URL: https://github.com/apache/seatunnel/pull/11423#discussion_r3569434790
##########
docs/en/connectors/source/S3File.md:
##########
@@ -453,7 +413,7 @@ source {
transform {
# If you would like to get more information about how to configure seatunnel
and see full list of transform plugins,
- # please go to https://seatunnel.apache.org/docs/transforms
+ # please go to https://seatunnel.apache.org/docs/transform-v2
Review Comment:
Only two lines below, the docs now point to `/docs/transform-v2`, but the
rest of the documentation set still uses `/docs/transforms`. Unless the site
has already migrated, this looks like a broken/inconsistent link.
##########
docs/en/connectors/source/S3File.md:
##########
@@ -226,9 +224,33 @@ If you assign file type to `parquet` `orc`, schema option
not required, connecto
| common-options | | no | -
| Source plugin common parameters, please refer
to [Source Common Options](../common-options/source-common-options.md) for
details.
|
| quote_char | string | no | "
| A single character that encloses CSV fields,
allowing fields with commas, line breaks, or quotes to be read correctly.
|
| escape_char | string | no | -
| A single character that allows the quote or
other special characters to appear inside a CSV field without ending the field.
|
-| metalake_type | string | no | gravitino
| The type of metalake service, currently
supports `gravitino`.
|
-| recursive_file_scan | boolean | no | true
| Whether to scan subdirectories recursively.
If `false`, subdirectories will be ignored.
|
-| sort_files_by_modification_time | boolean | no | false
| Sort files by modification time in descending
order. Enable this when reading evolving schemas to ensure schema inference
uses the latest file.
|
+
+### fs.s3a.aws.credentials.provider [string]
+
+SeaTunnel delegates S3 authentication to Hadoop S3A. This option is passed to
Hadoop as `fs.s3a.aws.credentials.provider`.
+
+Supported values:
+
+```hocon
+fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
+```
+
+```hocon
+fs.s3a.aws.credentials.provider =
"com.amazonaws.auth.InstanceProfileCredentialsProvider"
+```
+
+Container credential providers (for example
`com.amazonaws.auth.DefaultAWSCredentialsProviderChain`,
`com.amazonaws.auth.ContainerCredentialsProvider`) are not supported by this
connector option currently.
+
+If you use SeaTunnel Zeta, make sure the required Hadoop/AWS jars are present
on the runtime classpath (see the Dependency section in this document). If you
use Spark/Flink, make sure the cluster classpath includes the same jars.
+
Review Comment:
The section says container credential providers are not supported, which
matches the connector’s enum validation, but it conflicts with the PR
description’s claim that `DefaultAWSCredentialsProviderChain` is the
recommended container pattern for S3File connectors. Also, even if users try to
set `fs.s3a.aws.credentials.provider` via `hadoop_s3_properties`, the connector
will overwrite it. Please align the PR description and clarify the override
behavior here to prevent confusing configurations.
##########
docs/zh/connectors/source/S3File.md:
##########
@@ -225,9 +223,41 @@ schema {
| common-options | | 否 | -
|
数据源插件通用参数，请参考[数据源通用选项](../common-options/source-common-options.md)了解详情。
|
| quote_char | string | 否 | "
| 用于包裹 CSV 字段的单字符，可保证包含逗号、换行符或引号的字段被正确解析。
|
| escape_char | string | 否 | -
| 用于在 CSV 字段内转义引号或其他特殊字符，使其不会结束字段。
|
-| metalake_type | string | 否 | gravitino
| Metalake 服务类型，目前支持 `gravitino`。
|
-| recursive_file_scan | boolean | 否 | true
| 是否递归扫描子目录。 如果设置为 `false`，将忽略子目录，仅扫描指定路径下的文件。
|
-| sort_files_by_modification_time | boolean | 否 | false |
是否按修改时间降序排序文件。启用此选项后，在读取不断演化的 schema 时可确保 schema 推断使用最新的文件。
|
+
+### fs.s3a.aws.credentials.provider [string]
+
+SeaTunnel 将 S3 认证委托给 Hadoop S3A，该参数会以 `fs.s3a.aws.credentials.provider` 的形式透传给
Hadoop。
+
+支持的取值：
+
+```hocon
+fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
+```
+
+```hocon
+fs.s3a.aws.credentials.provider =
"com.amazonaws.auth.InstanceProfileCredentialsProvider"
+```
+
+容器凭据相关的 provider（例如
`com.amazonaws.auth.DefaultAWSCredentialsProviderChain`、`com.amazonaws.auth.ContainerCredentialsProvider`）当前不受该
connector 参数支持。
+
+如果使用 SeaTunnel Zeta，请确保运行时 classpath 中已包含 Hadoop/AWS 相关 jar（可参考本文档的 Dependency
章节）。如果使用 Spark/Flink，请确保集群 classpath 中也包含对应 jar。
+
Review Comment:
这里说明容器凭据 provider 不受支持与代码实现一致（该参数在 connector 内部做了枚举校验），但与 PR 描述中“对 S3File
连接器推荐 DefaultAWSCredentialsProviderChain”的表述不一致。另外即使用户尝试通过
`hadoop_s3_properties` 传入 `fs.s3a.aws.credentials.provider`，也会被 connector
覆盖。建议在此处明确覆盖行为，并同步修正 PR 描述以免误导。
##########
docs/en/connectors/sink/S3File.md:
##########
@@ -163,6 +162,35 @@ hadoop_s3_properties {
}
```
+### fs.s3a.aws.credentials.provider [string]
+
+SeaTunnel delegates S3 authentication to Hadoop S3A. This option is passed to
Hadoop as `fs.s3a.aws.credentials.provider`.
+
+Supported values:
+
+```
+fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
+```
+
+```
+fs.s3a.aws.credentials.provider =
"com.amazonaws.auth.InstanceProfileCredentialsProvider"
+```
+
+Container credential providers (for example
`com.amazonaws.auth.DefaultAWSCredentialsProviderChain`,
`com.amazonaws.auth.ContainerCredentialsProvider`) are not supported by this
connector option currently.
Review Comment:
This doc says container credential providers are unsupported, but doesn’t
mention that the connector will overwrite `fs.s3a.aws.credentials.provider`
even if users set it inside `hadoop_s3_properties`. Since this is a common
workaround attempt, explicitly calling out that it won’t work would reduce
confusion.
##########
docs/en/connectors/sink/S3File.md:
##########
@@ -374,12 +402,12 @@ source {
}
}
# If you would like to get more information about how to configure seatunnel
and see full list of source plugins,
- # please go to https://seatunnel.apache.org/docs/connectors/source
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source
}
transform {
# If you would like to get more information about how to configure seatunnel
and see full list of transform plugins,
- # please go to https://seatunnel.apache.org/docs/transforms
+ # please go to https://seatunnel.apache.org/docs/transform-v2
Review Comment:
The example footer link was changed to `/docs/transform-v2`, but other docs
still use `/docs/transforms`. This makes the S3File page inconsistent and may
break navigation.
##########
docs/en/connectors/sink/S3File.md:
##########
@@ -374,12 +402,12 @@ source {
}
}
# If you would like to get more information about how to configure seatunnel
and see full list of source plugins,
- # please go to https://seatunnel.apache.org/docs/connectors/source
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source
Review Comment:
The example footer link was changed to `/docs/connector-v2/source`, but
other connector docs still point to `/docs/connectors/source`. Unless the
website structure has already changed globally, this creates an inconsistent
(and likely broken) link.
##########
docs/zh/connectors/sink/S3File.md:
##########
@@ -369,13 +397,13 @@ source {
}
}
# 如果您想了解更多关于如何配置SeaTunnel以及查看完整的源插件列表，
-# 请访问 https://seatunnel.apache.org/docs/connectors/source
+# 请访问 https://seatunnel.apache.org/docs/connector-v2/source
source {
}
transform {
# 如果您想了解更多关于如何配置SeaTunnel以及查看完整的转换插件列表，
- # 请访问 https://seatunnel.apache.org/docs/transforms
+ # 请访问 https://seatunnel.apache.org/docs/transform-v2
Review Comment:
示例末尾的转换插件链接改成了 `/docs/transform-v2`，但仓库内其它文档仍使用
`/docs/transforms`。建议保持一致，避免导航失效。
##########
docs/en/engines/zeta/checkpoint-storage.md:
##########
@@ -159,9 +159,31 @@ seatunnel:
storage.type: s3
s3.bucket: your-bucket
fs.s3a.endpoint: your-endpoint
- fs.s3a.aws.credentials.provider:
org.apache.hadoop.fs.s3a.InstanceProfileCredentialsProvider
+ fs.s3a.aws.credentials.provider:
com.amazonaws.auth.InstanceProfileCredentialsProvider
```
+If you run SeaTunnel in container environments (Kubernetes/ECS/EKS) and want
to avoid hardcoding long-lived access keys in config, you can use the AWS SDK
v1 default credential chain and let the runtime provide short-lived credentials
(task role, IRSA, etc.):
+
Review Comment:
The PR description says `ContainerCredentialsProvider` can be used when
available on the runtime classpath, but this checkpoint-storage section only
mentions `DefaultAWSCredentialsProviderChain`. Adding an explicit note/example
for `com.amazonaws.auth.ContainerCredentialsProvider` would better cover ECS
task-role setups and align with the stated scope.
##########
docs/zh/engines/zeta/checkpoint-storage.md:
##########
@@ -157,9 +157,31 @@ seatunnel:
storage.type: s3
s3.bucket: your-bucket
fs.s3a.endpoint: your-endpoint
- fs.s3a.aws.credentials.provider:
org.apache.hadoop.fs.s3a.InstanceProfileCredentialsProvider
+ fs.s3a.aws.credentials.provider:
com.amazonaws.auth.InstanceProfileCredentialsProvider
```
+如果您在容器环境（Kubernetes/ECS/EKS）中运行 SeaTunnel，并希望避免在配置中硬编码长期 AK/SK，可以使用 AWS SDK v1
的默认凭据链，从运行环境获取短期凭据（task role、IRSA 等）：
+
Review Comment:
PR 描述提到在运行时 classpath 可用时可以使用 `ContainerCredentialsProvider`，但这里的
checkpoint-storage 文档只提到了 `DefaultAWSCredentialsProviderChain`。建议补充一句对
`com.amazonaws.auth.ContainerCredentialsProvider` 的说明/示例，以覆盖 ECS task role 场景并与
PR 目标保持一致。
##########
docs/en/connectors/sink/S3File.md:
##########
@@ -146,7 +146,6 @@ If write to `csv`, `text` file type, All column will be
string.
| enable_header_write | boolean | no | false
| Only used when file_format_type is
text,csv.<br/> false:don't write header,true:write header.
|
| encoding | string | no | "UTF-8"
| Only used when file_format_type is
json,text,csv,xml.
|
| merge_update_event | boolean | no | false
| Only used when file_format_type is
canal_json,debezium_json or maxwell_json. When value is true, the UPDATE_AFTER
and UPDATE_BEFORE event will be merged into UPDATE event data |
-| schema_evolution_enabled | boolean | no | false
| Enable schema evolution support for CDC pipelines.
When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to
the sink without a job restart. Not supported for binary format. |
Review Comment:
`schema_evolution_enabled` is still supported by the file sinks (see
`FileBaseSinkOptions.SCHEMA_EVOLUTION_ENABLED`), but it was removed from this
S3File sink options table (and the detailed section was also deleted). This
makes the S3File sink docs incorrect/incomplete for CDC pipelines.
##########
docs/zh/connectors/sink/S3File.md:
##########
@@ -160,6 +159,35 @@ hadoop_s3_properties {
}
```
+### fs.s3a.aws.credentials.provider [string]
+
+SeaTunnel 将 S3 认证委托给 Hadoop S3A，该参数会以 `fs.s3a.aws.credentials.provider` 的形式透传给
Hadoop。
+
+支持的取值：
+
+```
+fs.s3a.aws.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
+```
+
+```
+fs.s3a.aws.credentials.provider =
"com.amazonaws.auth.InstanceProfileCredentialsProvider"
+```
+
+容器凭据相关的 provider（例如
`com.amazonaws.auth.DefaultAWSCredentialsProviderChain`、`com.amazonaws.auth.ContainerCredentialsProvider`）当前不受该
connector 参数支持。
Review Comment:
这里说明容器凭据 provider 不受支持，但没有说明一个容易踩坑的点：即使用户把 `fs.s3a.aws.credentials.provider`
写到 `hadoop_s3_properties` 里，也会被 connector 覆盖。建议补充该说明，避免用户误以为可以通过透传绕过限制。
##########
docs/zh/connectors/sink/S3File.md:
##########
@@ -143,7 +143,6 @@ import ChangeLog from '../changelog/connector-file-s3.md';
| enable_header_write | boolean | 否 | false
| 仅当 file_format_type 为 text,csv 时使用。<br/>
false: 不写入表头, true: 写入表头。
|
| encoding | string | 否 | "UTF-8"
| 仅当 file_format_type 为 json,text,csv,xml
时使用。
|
| merge_update_event | boolean | 否 | false
|
仅当file_format_type为canal_json、debezium_json、maxwell_json.
|
-| schema_evolution_enabled | boolean | 否 | false
| 开启 Schema 演变支持，适用于 CDC 管道。为 true 时，来自上游的
ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
Review Comment:
`schema_evolution_enabled` 仍然是 File Sink 支持的参数（见
`FileBaseSinkOptions.SCHEMA_EVOLUTION_ENABLED`），但这里从 S3File Sink
的参数表中被移除了（对应的详细说明段落也被删除）。这会导致 CDC 管道用户无法发现该功能或误以为不支持。
##########
docs/zh/connectors/source/S3File.md:
##########
@@ -452,7 +420,7 @@ source {
transform {
# 如果您想获取有关如何配置seatunnel和查看转换插件完整列表的更多信息，
- # 请访问 https://seatunnel.apache.org/docs/transforms
+ # 请访问 https://seatunnel.apache.org/docs/transform-v2
Review Comment:
这里将链接改成了 `/docs/transform-v2`，但仓库内绝大多数文档仍使用
`/docs/transforms`。除非站点已完成迁移，否则该链接可能失效/不一致。
##########
docs/en/connectors/sink/S3File.md:
##########
@@ -407,7 +435,7 @@ sink {
}
}
# If you would like to get more information about how to configure seatunnel
and see full list of sink plugins,
- # please go to https://seatunnel.apache.org/docs/connectors/sink
+ # please go to https://seatunnel.apache.org/docs/connector-v2/sink
Review Comment:
The example footer link was changed to `/docs/connector-v2/sink`, but other
connector docs still point to `/docs/connectors/sink`. Unless the website
structure has changed everywhere, this should stay consistent.
##########
docs/zh/connectors/sink/S3File.md:
##########
@@ -369,13 +397,13 @@ source {
}
}
# 如果您想了解更多关于如何配置SeaTunnel以及查看完整的源插件列表，
-# 请访问 https://seatunnel.apache.org/docs/connectors/source
+# 请访问 https://seatunnel.apache.org/docs/connector-v2/source
Review Comment:
示例末尾的链接改成了 `/docs/connector-v2/source`，但仓库内其它连接器文档仍是
`/docs/connectors/source`。除非站点结构已全量迁移，否则这里会造成不一致（甚至失效）。
##########
docs/zh/connectors/sink/S3File.md:
##########
@@ -403,7 +431,7 @@ sink {
}
}
# 如果您想了解更多关于如何配置SeaTunnel以及查看完整的接收插件列表，
- # 请访问 https://seatunnel.apache.org/docs/connectors/sink
+ # 请访问 https://seatunnel.apache.org/docs/connector-v2/sink
Review Comment:
示例末尾的 sink 插件链接改成了 `/docs/connector-v2/sink`，但仓库内其它连接器文档仍使用
`/docs/connectors/sink`。除非站点结构已全量迁移，否则建议保持一致。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
