[
https://issues.apache.org/jira/browse/FLINK-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Lamar updated FLINK-10817:
-------------------------------
Description:
In order to use any given non-AWS s3 implementation backed by the presto s3
filesystem, it is necessary to set at least one configuration parameter in
flink-conf.yaml:
* presto.s3.endpoint: https://example.com
This appears to work as expected for hosted s3 alternatives.
In order to use a bring-your-own, self-hosted s3 alternative like
[minio|https://www.minio.io/], at least two configuration parameters are
required:
* presto.s3.endpoint: https://example.com
* presto.s3.path-style-access: true
However, the second path-style-access parameter doesn't work because the 0.185
version of presto doesn't support passing through that configuration option to
the hive s3 client.
To work around the issue, path-style-access can be forced on the s3 client by
using an IP address for the endpoint (instead of a hostname). Without this
workaround, flink attempts to use the virtualhost-style at
bucketname.example.com, which fails unless the expected DNS records exist.
To solve this problem and enable non-IP endpoints, upgrade the
[pom|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36]
to at least 0.186 which includes [this
commit|https://github.com/prestodb/presto/commit/0707f2f21a96d2fd30953fb3fa9a9a03f03d88bd]
Note that 0.213 is the latest presto release.
was:
In order to use any given non-AWS s3 implementation backed by the presto s3
filesystem, it is necessary to set at least one configuration parameter in
flink-conf.yaml:
* presto.s3.endpoint: https://example.com
This appears to work as expected for hosted s3 alternatives.
In order to use a bring-your-own, self-hosted s3 alternative like
[minio|https://www.minio.io/], at least two configuration parameters are
required:
* presto.s3.endpoint: https://example.com
* presto.s3.path-style-access: true
However, the second path-style-access parameter doesn't work because the 0.185
version of presto doesn't support passing through that configuration option to
the hive s3 client.
To work around the issue, path-style-access can be forced on the s3 client by
using an IP address for the endpoint (instead of a hostname). Without this
workaround, flink attempts to use the virtualhost-style at
bucketname.example.com, which fails unless the expected DNS records exist.
To solve this problem and enable non-IP endpoints, upgrade the
[pom|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36]
to at least 0.186 which includes [this
commit|[https://github.com/prestodb/presto/commit/0707f2f21a96d2fd30953fb3fa9a9a03f03d88bd.]
> Upgrade presto dependency to support path-style access
> ------------------------------------------------------
>
> Key: FLINK-10817
> URL: https://issues.apache.org/jira/browse/FLINK-10817
> Project: Flink
> Issue Type: Improvement
> Reporter: Adam Lamar
> Priority: Major
>
> In order to use any given non-AWS s3 implementation backed by the presto s3
> filesystem, it is necessary to set at least one configuration parameter in
> flink-conf.yaml:
> * presto.s3.endpoint: https://example.com
> This appears to work as expected for hosted s3 alternatives.
> In order to use a bring-your-own, self-hosted s3 alternative like
> [minio|https://www.minio.io/], at least two configuration parameters are
> required:
> * presto.s3.endpoint: https://example.com
> * presto.s3.path-style-access: true
> However, the second path-style-access parameter doesn't work because the
> 0.185 version of presto doesn't support passing through that configuration
> option to the hive s3 client.
> To work around the issue, path-style-access can be forced on the s3 client by
> using an IP address for the endpoint (instead of a hostname). Without this
> workaround, flink attempts to use the virtualhost-style at
> bucketname.example.com, which fails unless the expected DNS records exist.
> To solve this problem and enable non-IP endpoints, upgrade the
> [pom|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/pom.xml#L36]
> to at least 0.186 which includes [this
> commit|https://github.com/prestodb/presto/commit/0707f2f21a96d2fd30953fb3fa9a9a03f03d88bd]
> Note that 0.213 is the latest presto release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
