[
https://issues.apache.org/jira/browse/FLINK-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325617#comment-16325617
]
ASF GitHub Bot commented on FLINK-8432:
---------------------------------------
GitHub user jelmerk opened a pull request:
https://github.com/apache/flink/pull/5296
[FLINK-8432] [filesystem-connector] Add support for openstack's swift
filesystem
## What is the purpose of the change
Add support for OpenStack's cloud storage solution
## Brief change log
- Added new module below flink-filesystems
## Verifying this change
This change added tests and can be verified as follows:
- Added integration tests for simple reading and writing and listing
directories
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: don't know
- The runtime per-record code paths (performance sensitive): don't know
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: don't know
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? not documented
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jelmerk/flink openstack_fs_support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5296
----
commit d46696192cc57ba9df07fabf24c6d66538ec186d
Author: Jelmer Kuperus <jkuperus@...>
Date: 2018-01-14T09:42:02Z
[FLINK-8432] Add support for openstack's swift filesystem
----
> Add openstack swift filesystem
> ------------------------------
>
> Key: FLINK-8432
> URL: https://issues.apache.org/jira/browse/FLINK-8432
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
> Affects Versions: 1.4.0
> Reporter: Jelmer Kuperus
> Labels: features
>
> At ebay classifieds we started running our infrastructure on top of OpenStack.
> The openstack project comes with its own amazon-s3-like filesystem, known as
> Swift. It's built for scale and optimized for durability, availability, and
> concurrency across the entire data set. Swift is ideal for storing
> unstructured data that can grow without bound.
> We would really like to be able to use it within flink, as a sink or for
> storing savepoints etc
> I've prepared a pull request that adds support for it. It wraps the hadoop
> support for swift in a way that is very similar to the way the s3 connector
> works.
> You can find out more about the underlying hadoop implementation at
> https://hadoop.apache.org/docs/stable/hadoop-openstack/index.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
