[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Assignee: Steve Loughran (was: Thomas Demoor)
Status: Patch Available (was: Open)
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-010.patch,
> HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Attachment: HADOOP-9565-010.patch
Patch 010: resetting everything and starting again with nothing but an API to
query a filesystem for features. This is a WiP again, up for feedback.
* {{boolean FileSystem.hasFeature(Path, String)}} lets you see if the bit of an
FS under a path has a feature
* Feature names in {{FileSystemFeatures}} are pretty much a copy and paste of
{{org.apache.hadoop.fs.contract.ContractOptions}}.
* The base class mechanism for setting up feature resolution,
{{FeatureResolver}} is driven off XML configuration files; with a setup
mechanism which allows for subclasses to dynamically redefine the features,
such as when setting up a local FS.
All the work of defining features is already in the contract tests; I've just
pulled their declarations into the production hadoop-common, hadoop-hdfs-client
JARs, not the test JARs, and modified the two migrated FS's contracts to load
in the new XML file as well as the old one.
One big difference is in resolution of a feature: when a feature is looked for
it first looks for the schema-specific option
{{fs.hdfs.contract.supports-atomic-rename}} before falling back to look for
{{fs.contract.supports-atomic-rename}}, which is where a default can be
defined.
This allows for per-FS values to be set in core-default.xml,
{code}
# default: listings work
fs.contract.supports-consistent-listing = false
# but not S3A
fs.s3a.contract.supports-consistent-listing = false
{code}
And overridden in core-site.xml, as it'd be in a deployment against WDC's
servers
{code}
fs.s3a.contract.supports-consistent-listing = true
{code}
And it permits S3A per-bucket override for deployments where only some buckets
are consistent, such as "stevel-s3guard"
{code}
fs.s3a.bucket.stevel-s3guard.contract.supports-consistent-listing = true
{code}
Although the feature resolver is driven purely by the loaded hash of config
options, because the lookup is via the FS instance, it has the option of being
dynamic: per-path, on-demand evaluation rather than on construction probes for
OS type,
Also, the underlying feature resolver has a Java 8 optional API, so it can
return yes/no/don't know, the idea being "no" can be normative.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Thomas Demoor
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-010.patch,
> HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Status: Open (was: Patch Available)
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Thomas Demoor
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-9565:
--
Attachment: HADOOP-9565-008.patch
Rebased. Should revisit the features each file system supports.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Thomas Demoor
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pieter Reuse updated HADOOP-9565:
-
Attachment: HADOOP-9565-branch-2-007.patch
renamed file to HADOOP-9565-branch-2-007.patch so Hadoop QA can apply it.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pieter Reuse updated HADOOP-9565:
-
Attachment: (was: HADOOP-9565-007.patch)
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pieter Reuse updated HADOOP-9565:
-
Attachment: HADOOP-9565-007.patch
Uploaded patch version 7:
* Renamed UnsatisfiedSemanticsException to UnsatisfiedFeatureException
* Renamed ObjectStoreFeatures to ObjectStoreFeature and changed to an
enum-based approach
* added style improvements to FileOutputCommitter, since we're changing a lot
of lines throughout the class. Makes this patch prone to become non-applicable
when other patches change this class.
Chris, thanks for pointing out Steve will be enjoying a well earned break the
coming weeks. We will have patience w.r.t. his response.
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Labels: (was: BB2015-05-TBR)
Status: Patch Available (was: Open)
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Attachment: HADOOP-9565-006.patch
patch 006. Move off bitmask into string named features; various standard ones
defined.
I've also written a stub of a {{Put}} command; I'm wondering if we can do that
(and a Copy()), what could we do with an explicit committer for object stores
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Target Version/s: (was: )
Status: Open (was: Patch Available)
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pieter Reuse updated HADOOP-9565:
-
Assignee: Pieter Reuse (was: Thomas Demoor)
I've checked with [~Thomas Demoor] and I will work on a new patch addressing
the remarks of [~steve_l] and [~aartokhy].
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Demoor updated HADOOP-9565:
--
Attachment: HADOOP-9565-005.patch
Adjusted distcp: removed tempfile usage for object stores that support direct
concurrent writes. Evidently, this yields a 2x speedup .
AFAIK we have most things covered now?
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Thomas Demoor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Demoor updated HADOOP-9565:
--
Attachment: HADOOP-9565-004.patch
004 introduces an objectStore-aware FileOutputCommitter: it writes directly to
the outputPath if in-progress/failed writes are not visible on the object store.
[~jnp], [~anu], [~cnauroth] and [~jghoman]: this is the approach I think
HDFS-7240 can easily use this to avoid the rename issue.
The implementation optios were constrained by the fact that FileOutputCommitter
has a stable public interface:
* implementing the changes by subclassing was not possible
* the public / protected methods could not be changed. Often they are also
static (for use in MR1), further limiting options.
Some smaller stylistic changes (checkstyle, etc.) to the code in 003.patch
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Labels: BB2015-05-TBR
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
HADOOP-9565-003.patch, HADOOP-9565-004.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9565:
-
Labels: BB2015-05-TBR (was: )
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Labels: BB2015-05-TBR
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
HADOOP-9565-003.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Target Version/s: 3.0.0 (was: 2.7.0)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
HADOOP-9565-003.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Attachment: HADOOP-9565-003.patch
Patch -003
# renamed class to simply {{ObjectStore}}
# upped semantics mask to 64 bits long.
# pulled all semantic bitmasks to a {{StorageSemantics}} interface
# added the semantics of {{DISTINCT_FILES_AND_DIRECTORIES}},
{{WRITE_IN_PROGRESS_VISIBLE}} and {{WRITE_ON_FLUSH}}.
# marked up s3, s3n and swift. S3 really has a consistency model that is
instance-specific; need to make it something that you can declare in your fs
config options if it matters that much *and* you know what is offered.
# implemented use case of HADOOP-11525 ; handling of
{{WRITE_IN_PROGRESS_VISIBLE}} on a copy, so avoiding a PUT/GET/PUT/DELETE
sequence.
It'd be good to have a couple more uses of this method sorted out, so that we
can see how well it works...someone needs to write an output committer that
does the best it can given the semantics offered by the destination
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
HADOOP-9565-003.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Status: Patch Available (was: Open)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
HADOOP-9565-003.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Status: Open (was: Patch Available)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Component/s: fs/swift
fs/s3
Priority: Major (was: Minor)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Status: Open (was: Patch Available)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Attachment: HADOOP-9565-002.patch
patch -002; fixes missing header from test file
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Status: Patch Available (was: Open)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Attachment: HADOOP-9565-001.patch
This is the -001 patch, which defines a new class, {{abstract class
ObjectStoreFileSystem extends FileSystem }}, which is a marker interface. It
allows the FS Semantics to be queried, and requirements placed on an FS raise
an IOE if they aren't met. semantics include: consistency (create, delete,
list, update), atomicity (create, delete, rename...) and whether flush() does
anything.
This patch is just the class and tests, without any update to the FS
specification doc, and without reparenting the in-hadoop-codebase object
stores, which for swift:// and //s3? come with nearly no guarantees.
# Swift offers create consistency
# s3 create consistency varies a bit by region though.
# Azure does offer a consistent mode, so you can swap hdfs out for it.
# Netflix S3mper and AWS equivalent does improve s3 consistency, especially
metadata consistency
I don't expect apps to adjust their behaviour based on the guarantees, but
think some code could/should reject operations if the dest isn't suitable for
their actions (e.g. speculative operations committed by rename() require
atomic rename operations)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Target Version/s: 2.7.0
Affects Version/s: (was: 2.0.4-alpha)
2.6.0
Status: Patch Available (was: Open)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: HADOOP-9565-001.patch
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
---
Issue Type: Improvement (was: Sub-task)
Parent: (was: HADOOP-9361)
Add a Blobstore interface to add to blobstore FileSystems
-
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
