[
https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189624#comment-17189624
]
Arpit Agarwal edited comment on HDDS-4097 at 9/2/20, 6:30 PM:
--------------------------------------------------------------
Thanks for the doc, I will take a look.
However this is only partially related to the original proposal from Bharat.
The proposal here states:
# There is a set of use-cases where files are ingested via S3 and accessed via
HCFS. E.g. someone ingesting logs into Hive external tables via fluentd/S3. For
those use cases, the key names must look like valid paths and we need to ensure
they are checked and normalized appropriately and directory prefixes created.
# There is another set of pure object store use cases where the paths are
random strings and may have arbitrary characters including {{/}} or other
characters which are not valid in an FS path. Ingestion of such keys should be
successful and must not fail.
These are mutually exclusive use cases. In case #2, you cannot ingest invalid
paths while maintaining filesystem integrity. The point of providing a config
key or switch is to allow the user to choose, with the current default being
#2. This umbrella task is proposing to improve the first use case. What you are
proposing is for the second use case and it is a completely separate
discussion. So what's the overlap?
was (Author: arpitagarwal):
Thanks for the doc, I will take a look.
However this is only partially related to the original proposal from Bharat.
The proposal here states:
# There is a set of use-cases where files are ingested via S3 and accessed via
HCFS. E.g. someone ingesting logs into Hive external tables via fluentd/S3. For
those use cases, the key names must look like valid paths and we need to ensure
they are checked and normalized appropriately and directory prefixes created.
# There is another set of pure object store use cases where the paths are
random strings and may have arbitrary characters including {{/}} or other
characters which are not valid in an FS path. Ingestion of such keys should be
successful and must not fail.
These are mutually exclusive use cases. In case #2, you cannot ingest invalid
paths while maintaining filesystem integrity. The point of providing a config
key or switch is to allow the user to choose, with the current default being
#2. This umbrella task is proposing to fix the first use case. What you are
proposing is for the second use case. So what's the overlap?
> S3/Ozone Filesystem inter-op
> ----------------------------
>
> Key: HDDS-4097
> URL: https://issues.apache.org/jira/browse/HDDS-4097
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem
> path enabled.xlsx
>
>
> This Jira is to implement changes required to use Ozone buckets when data is
> ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial
> implementation for this is done as part of HDDS-3955. There are few API's
> which have missed the changes during the implementation of HDDS-3955.
> Attached design document which discusses each API, and what changes are
> required.
> Excel sheet has information about each API, from what all interfaces the OM
> API is used, and what changes are required for the API to support
> inter-operability.
> Note: The proposal for delete/rename is still under discussion, not yet
> finalized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
