[
https://issues.apache.org/jira/browse/BEAM-9094?focusedWorklogId=371154&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371154
]
ASF GitHub Bot logged work on BEAM-9094:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Jan/20 22:20
Start Date: 13/Jan/20 22:20
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #10560: BEAM-9094: Support
setting some options such as endpoint_url and credential infos for AWS S3
Filesystem in Python SDKs
URL: https://github.com/apache/beam/pull/10560#issuecomment-573900091
That's correct. As @lukecwik points out, initialization for filesystems is
performed in the filesystem's init method using pipeline options:
See hadoopfilesystem:
https://github.com/apache/beam/blob/d35195f4c40ff154ff0c4aa33f402acc10937e37/sdks/python/apache_beam/io/hadoopfilesystem.py#L100-L133
You could pass this with a new init method for s3filesystem that sets the
configuration for S3IO:
https://github.com/apache/beam/blob/d35195f4c40ff154ff0c4aa33f402acc10937e37/sdks/python/apache_beam/io/aws/s3filesystem.py
The Java filesystem does not seem to use a client that way, but it receives
the options via pipeline options:
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java#L105-L121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 371154)
Time Spent: 1h 10m (was: 1h)
> Support setting some options such as endpoint_url and credential infos for
> AWS S3 Filesystem in Python SDKs
> -----------------------------------------------------------------------------------------------------------
>
> Key: BEAM-9094
> URL: https://issues.apache.org/jira/browse/BEAM-9094
> Project: Beam
> Issue Type: Improvement
> Components: io-ideas
> Affects Versions: 2.19.0
> Reporter: Keunhyun Oh
> Assignee: Keunhyun Oh
> Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> AWS S3 File System is implemented in BEAM-2572.
> To use local s3 like minio, It is need to support setting some options such
> as endpoint_url and credential infos.
> In my idea, it can be implemented to use environment variables.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
