[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=551336&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-551336 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 11/Feb/21 15:06 Start Date: 11/Feb/21 15:06 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-777552648 > No one should do this. Only want it for stable tests. that and to challenge sqllite for dominance in the embedded-database market This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 551336) Time Spent: 11h 10m (was: 11h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 11h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550877&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550877 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 10/Feb/21 19:33 Start Date: 10/Feb/21 19:33 Worklog Time Spent: 10m Work Description: ndimiduk commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776962152 No one should do this. Only want it for stable tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550877) Time Spent: 11h (was: 10h 50m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 11h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550839 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 10/Feb/21 18:48 Start Date: 10/Feb/21 18:48 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776930067 @ndimiduk this will let you run HBase against local FS now as if you turn off the checksumming the WAL will be synced properly... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550839) Time Spent: 10h 50m (was: 10h 40m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550812 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 10/Feb/21 17:29 Start Date: 10/Feb/21 17:29 Worklog Time Spent: 10m Work Description: ndimiduk commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776881320 Thank you for this effort, @steveloughran ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550812) Time Spent: 10h 40m (was: 10.5h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550659 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 10/Feb/21 10:29 Start Date: 10/Feb/21 10:29 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2587: URL: https://github.com/apache/hadoop/pull/2587 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550659) Time Spent: 10.5h (was: 10h 20m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 10.5h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550572&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550572
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 10/Feb/21 05:15
Start Date: 10/Feb/21 05:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776448065
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 18s | | trunk passed |
| +1 :green_heart: | compile | 27m 42s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 23m 17s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 6s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 4s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 11s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 11m 36s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 24s | | the patch passed |
| +1 :green_heart: | compile | 20m 16s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 20m 16s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/4/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 generated 11 new + 2026 unchanged - 0
fixed = 2037 total (was 2026) |
| +1 :green_heart: | compile | 18m 2s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| -1 :x: | javac | 18m 2s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/4/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 11 new + 1919
unchanged - 0 fixed = 1930 total (was 1919) |
| -0 :warning: | checkstyle | 3m 55s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/4/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 1 new + 183 unchanged - 7 fixed = 184 total (was
190) |
| +1 :green_heart: | mvnsite | 6m 37s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/4/artifact/out/whitespace-eol.txt)
| The patch has 2 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 13m 16s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 7s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 12s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 12m 13s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 16s | | hadoop-common in the patch
passed. |
|
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550282 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 09/Feb/21 15:12 Start Date: 09/Feb/21 15:12 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776011799 done the final comments and then rebased; unless yetus really blows up will merge in This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550282) Time Spent: 10h 10m (was: 10h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 10h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550274 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 09/Feb/21 15:01 Start Date: 09/Feb/21 15:01 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#discussion_r572958871 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,1002 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in the client -the `hsync()` and `hflush()` operations the actions +which propagate the data to be visible to other readers of the file and/or +made durable. + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550273&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550273 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 09/Feb/21 14:59 Start Date: 09/Feb/21 14:59 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#discussion_r572957621 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,1002 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in the client -the `hsync()` and `hflush()` operations the actions +which propagate the data to be visible to other readers of the file and/or +made durable. + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. Review comment: up to the implementation. Close() must pass all its data to the shared FS. Now, if you want some fun, look at [NFS client side caching](https://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch07_04.htm). Dates from the era of diskless sun workstations and was optimised for short lived files which would only be used by the workstations, so copying over a 1MB/s shared ethernet to an even slower shared HDD would hurt the rest of the cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 550273) Time Spent: 9h 50m (was: 9h 40m) > Add OutputStream + Syncable to the Filesystem Specification > -
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550269 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 09/Feb/21 14:50 Start Date: 09/Feb/21 14:50 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#discussion_r572948194 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,1002 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in the client -the `hsync()` and `hflush()` operations the actions +which propagate the data to be visible to other readers of the file and/or +made durable. + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=550116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-550116
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 09/Feb/21 09:04
Start Date: 09/Feb/21 09:04
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#2587:
URL: https://github.com/apache/hadoop/pull/2587#discussion_r572182556
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/StoreImplementationUtils.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.StreamCapabilities.HFLUSH;
+import static org.apache.hadoop.fs.StreamCapabilities.HSYNC;
+
+/**
+ * Utility classes to help implementing filesystems and streams.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public final class StoreImplementationUtils {
+
+ private StoreImplementationUtils() {
+ }
+
+ /**
+ * Check the probe capability being for {@link StreamCapabilities#HSYNC}
+ * or {@link StreamCapabilities#HFLUSH}
+ * {@code Syncable.hsync()} and {@code Syncable.hflush()} functionality.
+ * @param capability capability string.
+ * @return true if either refers to one of the Syncable operations.
+ */
+ public static boolean isProbeForSyncable(String capability) {
+return capability.equalsIgnoreCase(HSYNC) ||
+capability.equalsIgnoreCase(HFLUSH);
+ }
+
+ /**
+ * Probe for an object having a capability; returns true
+ * if the stream implements {@link StreamCapabilities} and its
+ * {@code hasCapabilities()} method returns true for the capability.
+ * This is a package private method intended to provided a common
+ * implementation for input and output streams.
+ * {@link StreamCapabilities#hasCapability(String)} call is for public use.
+ * @param object object to probe.
+ * @param capability capability to probe for
+ * @return true if the object implements stream capabilities and
+ * declares that it supports the capability.
+ */
+ static boolean objectHasCapability(Object object, String capability) {
+if (object instanceof StreamCapabilities) {
+ return ((StreamCapabilities) object).hasCapability(capability);
+}
+return false;
+ }
+
+ /**
+ * Probe for an output stream having a capability; returns true
+ * if the stream implements {@link StreamCapabilities} and its
+ * {@code hasCapabilities()} method returns true for the capability.
+ * @param out output stream
+ * @param capability capability to probe for
+ * @return true if the stream declares that it supports the capability.
+ */
+ public static boolean hasCapability(OutputStream out, String capability) {
+return objectHasCapability(out, capability);
+ }
+
+ /**
+ * Probe for an input stream having a capability; returns true
+ * if the stream implements {@link StreamCapabilities} and its
+ * {@code hasCapabilities()} method returns true for the capability.
+ * @param out output stream
Review comment:
input stream
##
File path:
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md
##
@@ -0,0 +1,1002 @@
+
+
+
+
+# Output: `OutputStream`, `Syncable` and `StreamCapabilities`
+
+## Introduction
+
+This document covers the Output Streams within the context of the
+[Hadoop File System Specification](index.html).
+
+It uses the filesystem model defined in [A Model of a Hadoop
Filesystem](model.html)
+with the notation defined in [notation](Notation.md).
+
+The target audiences are:
+1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces,
+this document clarifies how it is implemented in HDFS and elsewhere.
+The Hadoop-specific interfaces `Syncable
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=549544&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549544 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 08/Feb/21 12:30 Start Date: 08/Feb/21 12:30 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-775113631 FWIW, here are the IOStats logged on the runs ABFS (which supports hsync) ``` .dfs.core.windows.net/stevel-testing/test/testSyncable?action=flush&retainUncommittedData=false&position=2&close=true&timeout=90 2021-02-08 12:27:48,642 INFO [JUnit-testSyncable]: contract.AbstractContractCreateTest (AbstractContractCreateTest.java:validateSyncableSemantics(586)) - IOStatistics counters=((queue_shrunk_ops=1) (time_spent_task_wait.failures=0) (write_current_buffer_ops=2) (bytes_upload_failed=0) (bytes_upload=2) (time_spent_on_put_request.failures=0) (time_spent_on_put_request=2) (time_spent_task_wait=0) (bytes_upload_successfully=2)); gauges=(); minimums=((time_spent_task_wait.failures.min=-1) (time_spent_on_put_request.min=28) (time_spent_task_wait.min=-1) (time_spent_on_put_request.failures.min=-1)); maximums=((time_spent_on_put_request.failures.max=-1) (time_spent_task_wait.max=-1) (time_spent_on_put_request.max=36) (time_spent_task_wait.failures.max=-1)); means=((time_spent_task_wait.mean=(samples=0, sum=0, mean=0.)) (time_spent_on_put_request.failures.mean=(samples=0, sum=0, mean=0.)) (time_spent_task_wait.failures.mean=(samples=0, sum=0, mean=0.)) (time_spent_on_put_request.mean=(samples=2, sum=64, mean=32.))); ``` s3a ``` 2021-02-08 12:29:05,294 [JUnit-testSyncable] WARN contract.AbstractContractCreateTest (AbstractContractCreateTest.java:validateSyncableSemantics(575)) - Output file was not created; this is an object store with different visibility semantics 2021-02-08 12:29:05,500 [JUnit-testSyncable] INFO contract.AbstractContractCreateTest (AbstractContractCreateTest.java:validateSyncableSemantics(586)) - IOStatistics counters=((stream_write_bytes=2) (stream_write_queue_duration=0) (action_executor_acquired=1) (stream_write_block_uploads=1) (stream_write_exceptions=0) (stream_write_exceptions_completing_upload=0) (stream_write_total_time=0) (stream_write_total_data=2) (action_executor_acquired.failures=0)); gauges=((stream_write_block_uploads_data_pending=0) (stream_write_block_uploads_pending=1)); minimums=((action_executor_acquired.failures.min=-1) (action_executor_acquired.min=0)); maximums=((action_executor_acquired.max=0) (action_executor_acquired.failures.max=-1)); means=((action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.)) (action_executor_acquired.mean=(samples=1, sum=0, mean=0.))); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 549544) Time Spent: 9h 20m (was: 9h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=549510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549510
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 08/Feb/21 11:00
Start Date: 08/Feb/21 11:00
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2587:
URL: https://github.com/apache/hadoop/pull/2587#discussion_r571955452
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
##
@@ -137,8 +139,13 @@ public void initialize(URI uri, Configuration conf) throws
IOException {
STREAM_READ_SKIP_BYTES)
.build();
+/** Reference to the bytes read counter for slightly faster counting. */
+private final AtomicLong bytesRead;
+
public LocalFSFileInputStream(Path f) throws IOException {
fis = new FileInputStream(pathToFile(f));
+ bytesRead = ioStatistics.getCounterReference(
Review comment:
this is a followon to IOStats...it was actually blocked on the buffer
between row local fs output and FSDataOutput:
`org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream` - needed
for hsync passthrough to the File output stream. We're now generating IOStats
for the local FS too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 549510)
Time Spent: 9h 10m (was: 9h)
> Add OutputStream + Syncable to the Filesystem Specification
> ---
>
> Key: HADOOP-13327
> URL: https://issues.apache.org/jira/browse/HADOOP-13327
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch,
> HADOOP-13327-branch-2-001.patch
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Write down what a Filesystem output stream should do. While core the API is
> defined in Java, that doesn't say what's expected about visibility,
> durability, etc —and Hadoop Syncable interface is entirely ours to define.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=546522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-546522
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 03/Feb/21 01:13
Start Date: 03/Feb/21 01:13
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#discussion_r568481286
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+ public void testSyncable() throws Throwable {
+describe("test declared and actual Syncable behaviors");
+FileSystem fs = getFileSystem();
+boolean supportsFlush = isSupported(SUPPORTS_HFLUSH);
+boolean supportsSync = isSupported(SUPPORTS_HSYNC);
+
+validateSyncableSemantics(fs, supportsSync, supportsFlush);
+ }
+
+ /**
+ * Validate the semantics of syncable.
+ * @param fs filesystem
+ * @param supportsSync sync is present
+ * @param supportsFlush flush is present.
+ * @throws IOException failure
+ */
+ protected void validateSyncableSemantics(final FileSystem fs,
+ final boolean supportsSync, final boolean supportsFlush) throws
IOException {
+Path path = methodPath();
+LOG.info("Expecting files under {} to have supportsSync={}"
++ " and supportsFlush={}",
+path, supportsSync, supportsFlush);
+
+
+try (FSDataOutputStream out = fs.create(path, true)) {
+ LOG.info("Created output stream {}", out);
+
+ // probe stream for support for flush/sync, whose capabilities
+ // of supports/does not support must match what is expected
+ String[] hflushCapabilities = {
+ StreamCapabilities.HFLUSH
+ };
+ String[] hsyncCapabilities = {
+ StreamCapabilities.HSYNC
+ };
+ if (supportsFlush) {
+assertCapabilities(out, hflushCapabilities, null);
+ } else {
+assertCapabilities(out, null, hflushCapabilities);
+ }
+ if (supportsSync) {
+assertCapabilities(out, hsyncCapabilities, null);
+ } else {
+assertCapabilities(out, null, hsyncCapabilities);
+ }
+ out.write('a');
+ try {
+out.hflush();
+if (!supportsFlush) {
+ // hsync not ignored
Review comment:
little confusing comment, maybe we can change this to "hysc tests not
ignored"?
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
##
@@ -137,8 +139,13 @@ public void initialize(URI uri, Configuration conf) throws
IOException {
STREAM_READ_SKIP_BYTES)
.build();
+/** Reference to the bytes read counter for slightly faster counting. */
+private final AtomicLong bytesRead;
+
public LocalFSFileInputStream(Path f) throws IOException {
fis = new FileInputStream(pathToFile(f));
+ bytesRead = ioStatistics.getCounterReference(
Review comment:
No issues here, just confirming if these changes aren't part of a
different PR by mistake(IOStats PR).
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+ public void testSyncable() throws Throwable {
+describe("test declared and actual Syncable behaviors");
+FileSystem fs = getFileSystem();
+boolean supportsFlush = isSupported(SUPPORTS_HFLUSH);
+boolean supportsSync = isSupported(SUPPORTS_HSYNC);
+
+validateSyncableSemantics(fs, supportsSync, supportsFlush);
+ }
+
+ /**
+ * Validate the semantics of syncable.
+ * @param fs filesystem
+ * @param supportsSync sync is present
+ * @param supportsFlush flush is present.
+ * @throws IOException failure
+ */
+ protected void validateSyncableSemantics(final FileSystem fs,
+ final boolean supportsSync, final boolean supportsFlush) throws
IOException {
+Path path = methodPath();
+LOG.info("Expecting files under {} to have supportsSync={}"
++ " and supportsFlush={}",
+path, supportsSync, supportsFlush);
+
Review comment:
nit: blank line.
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=545988&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545988 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 02/Feb/21 11:40 Start Date: 02/Feb/21 11:40 Worklog Time Spent: 10m Work Description: mehakmeet commented on a change in pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#discussion_r568535248 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/localfs/TestLocalFSContractCreate.java ## @@ -18,7 +18,11 @@ package org.apache.hadoop.fs.contract.localfs; +import org.junit.Test; + import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; Review comment: unused import. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545988) Time Spent: 8h 50m (was: 8h 40m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 8h 50m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=545943&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545943
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 10:36
Start Date: 02/Feb/21 10:36
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#discussion_r568481286
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+ public void testSyncable() throws Throwable {
+describe("test declared and actual Syncable behaviors");
+FileSystem fs = getFileSystem();
+boolean supportsFlush = isSupported(SUPPORTS_HFLUSH);
+boolean supportsSync = isSupported(SUPPORTS_HSYNC);
+
+validateSyncableSemantics(fs, supportsSync, supportsFlush);
+ }
+
+ /**
+ * Validate the semantics of syncable.
+ * @param fs filesystem
+ * @param supportsSync sync is present
+ * @param supportsFlush flush is present.
+ * @throws IOException failure
+ */
+ protected void validateSyncableSemantics(final FileSystem fs,
+ final boolean supportsSync, final boolean supportsFlush) throws
IOException {
+Path path = methodPath();
+LOG.info("Expecting files under {} to have supportsSync={}"
++ " and supportsFlush={}",
+path, supportsSync, supportsFlush);
+
+
+try (FSDataOutputStream out = fs.create(path, true)) {
+ LOG.info("Created output stream {}", out);
+
+ // probe stream for support for flush/sync, whose capabilities
+ // of supports/does not support must match what is expected
+ String[] hflushCapabilities = {
+ StreamCapabilities.HFLUSH
+ };
+ String[] hsyncCapabilities = {
+ StreamCapabilities.HSYNC
+ };
+ if (supportsFlush) {
+assertCapabilities(out, hflushCapabilities, null);
+ } else {
+assertCapabilities(out, null, hflushCapabilities);
+ }
+ if (supportsSync) {
+assertCapabilities(out, hsyncCapabilities, null);
+ } else {
+assertCapabilities(out, null, hsyncCapabilities);
+ }
+ out.write('a');
+ try {
+out.hflush();
+if (!supportsFlush) {
+ // hsync not ignored
Review comment:
little confusing comment, maybe we can change this to "hysc tests not
ignored"?
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
##
@@ -137,8 +139,13 @@ public void initialize(URI uri, Configuration conf) throws
IOException {
STREAM_READ_SKIP_BYTES)
.build();
+/** Reference to the bytes read counter for slightly faster counting. */
+private final AtomicLong bytesRead;
+
public LocalFSFileInputStream(Path f) throws IOException {
fis = new FileInputStream(pathToFile(f));
+ bytesRead = ioStatistics.getCounterReference(
Review comment:
No issues here, just confirming if these changes aren't part of a
different PR by mistake(IOStats PR).
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+ public void testSyncable() throws Throwable {
+describe("test declared and actual Syncable behaviors");
+FileSystem fs = getFileSystem();
+boolean supportsFlush = isSupported(SUPPORTS_HFLUSH);
+boolean supportsSync = isSupported(SUPPORTS_HSYNC);
+
+validateSyncableSemantics(fs, supportsSync, supportsFlush);
+ }
+
+ /**
+ * Validate the semantics of syncable.
+ * @param fs filesystem
+ * @param supportsSync sync is present
+ * @param supportsFlush flush is present.
+ * @throws IOException failure
+ */
+ protected void validateSyncableSemantics(final FileSystem fs,
+ final boolean supportsSync, final boolean supportsFlush) throws
IOException {
+Path path = methodPath();
+LOG.info("Expecting files under {} to have supportsSync={}"
++ " and supportsFlush={}",
+path, supportsSync, supportsFlush);
+
Review comment:
nit: blank line.
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
##
@@ -436,4 +447,110 @@ private void createFile(Path path) throws IOException {
writeDataset(fs, path, data, data.length, 1024 * 1024,
true);
}
+
+ @Test
+
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=544282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-544282 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 29/Jan/21 14:10 Start Date: 29/Jan/21 14:10 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-769827209 checkstyles and whitespace. The javac changes are all from deprecating hflush capability This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 544282) Time Spent: 8.5h (was: 8h 20m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 8.5h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=544281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-544281
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 29/Jan/21 14:09
Start Date: 29/Jan/21 14:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-768757128
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 26s | | trunk passed |
| +1 :green_heart: | compile | 27m 5s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 23m 20s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 34s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 16s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 16s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 39s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 19m 52s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 generated 11 new + 2037 unchanged - 0
fixed = 2048 total (was 2037) |
| +1 :green_heart: | compile | 17m 51s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| -1 :x: | javac | 17m 51s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 11 new + 1930
unchanged - 0 fixed = 1941 total (was 1930) |
| -0 :warning: | checkstyle | 3m 48s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 2 new + 182 unchanged - 7 fixed = 184 total (was
189) |
| +1 :green_heart: | mvnsite | 5m 45s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 13m 50s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 29s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 10m 49s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 14s | | hadoop-common in the patch
passed.
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=543744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-543744
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 28/Jan/21 16:48
Start Date: 28/Jan/21 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-764828144
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | | trunk passed |
| +1 :green_heart: | compile | 20m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 5m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 50s | | the patch passed |
| +1 :green_heart: | compile | 21m 44s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 21m 44s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2037 unchanged - 0
fixed = 2049 total (was 2037) |
| +1 :green_heart: | compile | 18m 41s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 18m 41s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1930
unchanged - 0 fixed = 1942 total (was 1930) |
| -0 :warning: | checkstyle | 2m 39s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 182 unchanged - 7 fixed = 186 total (was
189) |
| +1 :green_heart: | mvnsite | 5m 44s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 35s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=543703&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-543703
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 28/Jan/21 15:53
Start Date: 28/Jan/21 15:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-754200640
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 12s | | trunk passed |
| +1 :green_heart: | compile | 20m 4s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 15s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 27s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 3s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 4m 57s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 47s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 47s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 31s | | the patch passed |
| +1 :green_heart: | compile | 26m 52s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 26m 52s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2038 unchanged - 0
fixed = 2050 total (was 2038) |
| +1 :green_heart: | compile | 24m 4s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 24m 4s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1931
unchanged - 0 fixed = 1943 total (was 1931) |
| -0 :warning: | checkstyle | 3m 28s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 128 unchanged - 7 fixed = 132 total (was
135) |
| +1 :green_heart: | mvnsite | 6m 6s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 19m 15s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 10s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=543332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-543332
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 28/Jan/21 02:43
Start Date: 28/Jan/21 02:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-768757128
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 26s | | trunk passed |
| +1 :green_heart: | compile | 27m 5s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 23m 20s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 34s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 16s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 16s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 39s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 19m 52s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 generated 11 new + 2037 unchanged - 0
fixed = 2048 total (was 2037) |
| +1 :green_heart: | compile | 17m 51s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| -1 :x: | javac | 17m 51s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 11 new + 1930
unchanged - 0 fixed = 1941 total (was 1930) |
| -0 :warning: | checkstyle | 3m 48s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 2 new + 182 unchanged - 7 fixed = 184 total (was
189) |
| +1 :green_heart: | mvnsite | 5m 45s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/3/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 13m 50s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 29s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 10m 49s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 14s | | hadoop-common in the patch
passed. |
|
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=543269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-543269 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 28/Jan/21 01:38 Start Date: 28/Jan/21 01:38 Worklog Time Spent: 10m Work Description: joshelser commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-768714175 > you are still going to need some robust storage, RAID-1+ or similar, but we are getting sync all the way through, and you can query the streams to make sure they say they support it -including local fs. Yep, for sure. This is a nice improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 543269) Time Spent: 7h 40m (was: 7.5h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 7h 40m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=542373&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-542373
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 26/Jan/21 19:47
Start Date: 26/Jan/21 19:47
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-767785031
checkstyle nits
```
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java:512:
if (c == -1) {:24: Must have at least one statement. [EmptyBlock]
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:70:import
org.apache.hadoop.util.StringUtils;:8: Unused import -
org.apache.hadoop.util.StringUtils. [UnusedImports]
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobAppendStream.java:30:import
java.util.Locale;:8: Unused import - java.util.Locale. [UnusedImports]
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SyncableDataOutputStream.java:31:import
org.apache.hadoop.classification.InterfaceAudience;:1: Duplicate import to
line 28 - org.apache.hadoop.classification.InterfaceAudience. [RedundantImport]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 542373)
Time Spent: 7.5h (was: 7h 20m)
> Add OutputStream + Syncable to the Filesystem Specification
> ---
>
> Key: HADOOP-13327
> URL: https://issues.apache.org/jira/browse/HADOOP-13327
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch,
> HADOOP-13327-branch-2-001.patch
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Write down what a Filesystem output stream should do. While core the API is
> defined in Java, that doesn't say what's expected about visibility,
> durability, etc —and Hadoop Syncable interface is entirely ours to define.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=542372&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-542372 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 26/Jan/21 19:46 Start Date: 26/Jan/21 19:46 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-767784703 > Steve had mentioned to me that he was thinking of HBase and us being able to more safely put WALs onto file:// with these changes (not having to just disable the checks with LocalFileSystem). you are still going to need some robust storage, RAID-1+ or similar, but we are getting sync all the way through, and you can query the streams to make sure they say they support it -including local fs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 542372) Time Spent: 7h 20m (was: 7h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=541934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-541934 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 26/Jan/21 04:33 Start Date: 26/Jan/21 04:33 Worklog Time Spent: 10m Work Description: joshelser commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-766998412 Tagging a few HBase folks who may be interested in this as well, @ndimiduk @saintstack @Apache9 @busbey given the previous work on apache/hbase#1408 and apache/hbase#1597. Steve had mentioned to me that he was thinking of HBase and us being able to more safely put WALs onto file:// with these changes (not having to just disable the checks with LocalFileSystem). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 541934) Time Spent: 7h 10m (was: 7h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 7h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=541197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-541197 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 25/Jan/21 17:53 Start Date: 25/Jan/21 17:53 Worklog Time Spent: 10m Work Description: joshelser commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-766998412 Tagging a few HBase folks who may be interested in this as well, @ndimiduk @saintstack @Apache9 @busbey given the previous work on apache/hbase#1408 and apache/hbase#1597. Steve had mentioned to me that he was thinking of HBase and us being able to more safely put WALs onto file:// with these changes (not having to just disable the checks with LocalFileSystem). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 541197) Time Spent: 7h (was: 6h 50m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 7h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=539653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-539653
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 22/Jan/21 05:55
Start Date: 22/Jan/21 05:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-764828144
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | | trunk passed |
| +1 :green_heart: | compile | 20m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 5m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 50s | | the patch passed |
| +1 :green_heart: | compile | 21m 44s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 21m 44s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2037 unchanged - 0
fixed = 2049 total (was 2037) |
| +1 :green_heart: | compile | 18m 41s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 18m 41s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1930
unchanged - 0 fixed = 1942 total (was 1930) |
| -0 :warning: | checkstyle | 2m 39s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 182 unchanged - 7 fixed = 186 total (was
189) |
| +1 :green_heart: | mvnsite | 5m 44s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 35s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubunt
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=539231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-539231
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 21/Jan/21 17:54
Start Date: 21/Jan/21 17:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-764828144
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | | trunk passed |
| +1 :green_heart: | compile | 20m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 5m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 53s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 50s | | the patch passed |
| +1 :green_heart: | compile | 21m 44s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 21m 44s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2037 unchanged - 0
fixed = 2049 total (was 2037) |
| +1 :green_heart: | compile | 18m 41s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 18m 41s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1930
unchanged - 0 fixed = 1942 total (was 1930) |
| -0 :warning: | checkstyle | 2m 39s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 182 unchanged - 7 fixed = 186 total (was
189) |
| +1 :green_heart: | mvnsite | 5m 44s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 35s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubunt
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=530879&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530879
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 04/Jan/21 20:29
Start Date: 04/Jan/21 20:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-754200640
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 12s | | trunk passed |
| +1 :green_heart: | compile | 20m 4s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 15s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 27s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 3s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 4m 57s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 47s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 47s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 31s | | the patch passed |
| +1 :green_heart: | compile | 26m 52s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 26m 52s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2038 unchanged - 0
fixed = 2050 total (was 2038) |
| +1 :green_heart: | compile | 24m 4s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 24m 4s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1931
unchanged - 0 fixed = 1943 total (was 1931) |
| -0 :warning: | checkstyle | 3m 28s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 128 unchanged - 7 fixed = 132 total (was
135) |
| +1 :green_heart: | mvnsite | 6m 6s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 19m 15s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 10s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/1/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubunt
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=530711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530711 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 04/Jan/21 14:46 Start Date: 04/Jan/21 14:46 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2587: URL: https://github.com/apache/hadoop/pull/2587#issuecomment-754016813 * does not address final comments in that review * no ITest runs (I don't think it needs it; will look at again) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 530711) Time Spent: 6h 20m (was: 6h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 6h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=530691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530691 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 04/Jan/21 14:26 Start Date: 04/Jan/21 14:26 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #2102: URL: https://github.com/apache/hadoop/pull/2102 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 530691) Time Spent: 6h 10m (was: 6h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=530690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530690 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 04/Jan/21 14:26 Start Date: 04/Jan/21 14:26 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-754005159 Closing as #2587 is a rebased successor, one where the use of the new `BufferedIOStatisticsOutputStream` wrapper passes syncable through. With that change raw local FS does now support Syncable *correctly* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 530690) Time Spent: 6h (was: 5h 50m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 6h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=530688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530688 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 04/Jan/21 14:24 Start Date: 04/Jan/21 14:24 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #2587: URL: https://github.com/apache/hadoop/pull/2587 Specification of OutputStream and Syncable with * RawLocalFileSystem to implement Syncable * Consistent use of StreamCapabilities everywhere This is a rebase of #2102. Because the RawLocalOutputStream is now wrapped by `BufferedIOStatisticsOutputStream`, which does passthrough of stream capabilities and the Syncable API, the tests which were failing there should now work This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 530688) Time Spent: 5h 50m (was: 5h 40m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=516323&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516323
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 24/Nov/20 20:01
Start Date: 24/Nov/20 20:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-733202460
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 10s | | trunk passed |
| +1 :green_heart: | compile | 21m 23s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 5s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 51s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 21s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 5m 13s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 52s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 17s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 20m 58s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 20m 58s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/8/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2049 unchanged - 0
fixed = 2061 total (was 2049) |
| +1 :green_heart: | compile | 17m 50s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 17m 50s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/8/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1943
unchanged - 0 fixed = 1955 total (was 1943) |
| -0 :warning: | checkstyle | 2m 44s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/8/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 137 unchanged - 8 fixed = 142 total (was
145) |
| +1 :green_heart: | mvnsite | 5m 40s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/8/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 8s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 58s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/8/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubun
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=516156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516156 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 24/Nov/20 14:28 Start Date: 24/Nov/20 14:28 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-733008887 wow, that was a big failure. lets rebase and see what can "go away"., then look @ failures in detail This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516156) Time Spent: 5.5h (was: 5h 20m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514739
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 20/Nov/20 17:36
Start Date: 20/Nov/20 17:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-731304737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 54s | | trunk passed |
| +1 :green_heart: | compile | 27m 41s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 0s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 4m 43s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 42s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 16s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 45s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 20m 49s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/7/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 12 new + 2049 unchanged - 0
fixed = 2061 total (was 2049) |
| +1 :green_heart: | compile | 17m 56s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 17m 56s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/7/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 12 new + 1943
unchanged - 0 fixed = 1955 total (was 1943) |
| -0 :warning: | checkstyle | 2m 50s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/7/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 137 unchanged - 8 fixed = 142 total (was
145) |
| +1 :green_heart: | mvnsite | 5m 4s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/7/artifact/out/whitespace-eol.txt)
| The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 17m 46s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 56s |
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/7/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubun
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514227 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Nov/20 17:16 Start Date: 19/Nov/20 17:16 Worklog Time Spent: 10m Work Description: joshelser commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r527060336 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,952 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514199&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514199 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Nov/20 16:36 Start Date: 19/Nov/20 16:36 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r527029585 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,952 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514184 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Nov/20 16:14 Start Date: 19/Nov/20 16:14 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r527013218 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,952 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514170&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514170 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Nov/20 15:57 Start Date: 19/Nov/20 15:57 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r52640 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java ## @@ -32,7 +32,7 @@ import java.util.concurrent.Callable; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; - +This Review comment: heh, that's what comes of having speech recognition turned on This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 514170) Time Spent: 4h 40m (was: 4.5h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=514034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-514034 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Nov/20 09:56 Start Date: 19/Nov/20 09:56 Worklog Time Spent: 10m Work Description: mehakmeet commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r526702705 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java ## @@ -32,7 +32,7 @@ import java.util.concurrent.Callable; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; - +This Review comment: "This" should be removed. This might be the reason why Yetus is failing. ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,952 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stre
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=513726&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513726 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 18/Nov/20 19:23 Start Date: 18/Nov/20 19:23 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-729900720 yeah, lets cross ref the javadocs. "if you implement this you SHOULD declare that you support the capability" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513726) Time Spent: 4h 20m (was: 4h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=513718&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513718 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 18/Nov/20 19:12 Start Date: 18/Nov/20 19:12 Worklog Time Spent: 10m Work Description: joshelser commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-729894664 My only thought at this point is: are we creating a dichotomy by having HFLUSH be deprecated in `StreamCapabilities` but not deprecated in `Syncable`? The argument is that we would never have an implementor of hflush which doesn't also implement hsync. However, every implementor of hsync should have an implementation for hflush (even if that implementation is to just invoke hsync). Now that I've come to this, I think outputstream.md adequately does this. Do you think it's appropriate to have a reciprocal javadoc reference from `Syncable` back to `StreamCapabilities`? I was thinking, if I was writing against this API, I might not know to go looking for StreamCapabilities if I just start digging through the Syncable API. I would defer to your judgement here (if you think it's unnecessary, I am not concerned). This was a fantastic and thorough read. Thanks for taking the time to do it, Steve! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513718) Time Spent: 4h 10m (was: 4h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=513712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513712 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 18/Nov/20 19:09 Start Date: 18/Nov/20 19:09 Worklog Time Spent: 10m Work Description: joshelser commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r526350607 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,952 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are: +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The `Syncable` operations differ in their durability +guarantees, not visibility of data. + +### State of Stream and File System after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build()` within a filesystem `FS`, +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +FS' = FS where data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and File System after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)` within a filesystem `FS`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=513677&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513677 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 18/Nov/20 18:10 Start Date: 18/Nov/20 18:10 Worklog Time Spent: 10m Work Description: joshelser commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-729861275 Yikes. These got mis-filed in my mail. I'm sorry, Steve. > Declare that HSYNC is sufficient and the only one you should look for. If you implement HFLUSH and not HSYNC, you are of no use to applications. retain support for HFLUSH probe, but in the spec only mention it as of historical interest. My only worry with this is that, today, HBase is only running with HFlush and not HSync because of performance reasons. The last wag I remember was in the ballpark of a 30% performance hit for writes if we force hsync instead of hflush. Let me re-read your latest and see if my mind changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513677) Time Spent: 3h 50m (was: 3h 40m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=513668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513668
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 18/Nov/20 17:58
Start Date: 18/Nov/20 17:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-729854130
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 33m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 42s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 28m 3s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 24m 47s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 6s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 37s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 48s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 4m 42s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 44s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 13s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 19m 47s |
[/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.
|
| -1 :x: | javac | 19m 47s |
[/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.
|
| -1 :x: | compile | 17m 16s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -1 :x: | javac | 17m 16s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 2m 45s |
[/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/buildtool-patch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 28s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/6/artifact/out/whitespace-eol.txt)
| The patch has 3 line(s) that end in whitespace. Use git apply
--w
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=502412&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-502412
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 19/Oct/20 20:48
Start Date: 19/Oct/20 20:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-712433315
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 37m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 25m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 42m 8s | | trunk passed |
| +1 :green_heart: | compile | 21m 33s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 58s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 45s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 4m 45s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 44s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 53s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 14s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 19m 56s |
[/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 19m 56s |
[/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 17m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 17m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 2m 45s |
[/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/buildtool-patch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 28s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | white
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=502360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-502360 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 19/Oct/20 17:57 Start Date: 19/Oct/20 17:57 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-712339962 HSYNC is the only option needed; tag HFLUSH as deprecated and in the docs say its not useful; outputstream.md doc lists the standard options. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 502360) Time Spent: 3h 20m (was: 3h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=493502&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-493502 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 01/Oct/20 13:58 Start Date: 01/Oct/20 13:58 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696402075 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 9s | trunk passed | | +1 :green_heart: | compile | 23m 14s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 19m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 47s | trunk passed | | +1 :green_heart: | mvnsite | 4m 40s | trunk passed | | +1 :green_heart: | shadedclient | 23m 32s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 5s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 43s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 21m 10s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 21m 10s | the patch passed | | +1 :green_heart: | compile | 18m 10s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 18m 10s | the patch passed | | -0 :warning: | checkstyle | 2m 47s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 57s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 6s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 44s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 21s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 1s | hadoop-common in the patch passed. | | -1 :x: | unit | 101m 4s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 58s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 19s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 316m 36s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSnapshotCommands | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2102 | | Optiona
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=490802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490802 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 25/Sep/20 13:14 Start Date: 25/Sep/20 13:14 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-698236827 Thinking: * single capability, syncable, everything maps to * skip local FS test until we get the IOStats patch in, then re-enable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490802) Time Spent: 3h (was: 2h 50m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 3h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=490114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490114 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 24/Sep/20 09:42 Start Date: 24/Sep/20 09:42 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-698236827 Thinking: * single capability, syncable, everything maps to * skip local FS test until we get the IOStats patch in, then re-enable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490114) Time Spent: 2h 50m (was: 2h 40m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=488965&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488965 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 23/Sep/20 04:23 Start Date: 23/Sep/20 04:23 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696401829 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488965) Time Spent: 2h 40m (was: 2.5h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=488892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488892
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 23/Sep/20 04:17
Start Date: 23/Sep/20 04:17
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696757197
TestRawLocalContract create is failing until we do our own
BufferedOutputStream in passthrough of HADOOP-16830/ #2323 .
the HDFS checksum ones are about checksums not being found, which implies
they weren't being written. Which goes near output streams, doesn't it? Which
means there is a risk this is a genuine regression and not "hdfs tests being
flaky"
```
org.apache.hadoop.fs.PathIOException: `/striped/stripedFileChecksum1': Fail
to get block checksum for
LocatedStripedBlock{BP-1893408133-172.17.0.3-1600724641689:blk_-9223372036854775792_1001;
getBlockSize()=37748736; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:33131,DS-5c8811e5-39af-4967-b3a7-3b76f95b0317,DISK],
DatanodeInfoWithStorage[127.0.0.1:38603,DS-82754ee5-286a-402b-9861-b3ebbc149849,DISK],
DatanodeInfoWithStorage[127.0.0.1:44379,DS-080d6ce9-cb6a-4f80-b37a-e63ecc31d9bc,DISK],
DatanodeInfoWithStorage[127.0.0.1:34723,DS-c6c2aa4e-639b-4d85-9564-05631c8c5b79,DISK],
DatanodeInfoWithStorage[127.0.0.1:40061,DS-54eb3c16-b9ce-4a5d-a7a3-f33b635579b0,DISK],
DatanodeInfoWithStorage[127.0.0.1:45525,DS-3c9ac850-9273-4d2d-933c-2ac3b4b30308,DISK],
DatanodeInfoWithStorage[127.0.0.1:45537,DS-9b1d805c-3b0e-4c84-ad0e-f454817b6829,DISK],
DatanodeInfoWithStorage[127.0.0.1:44697,DS-e06897c8-c8a5-41d6-b5bc-57c7339bbf9b,DISK]];
indices=[1, 2, 3, 4, 5, 6, 7, 8]}
at
org.apache.hadoop.hdfs.FileChecksumHelper$StripedFileNonStripedChecksumComputer.checksumBlocks(FileChecksumHelper.java:640)
at
org.apache.hadoop.hdfs.FileChecksumHelper$FileChecksumComputer.compute(FileChecksumHelper.java:252)
at
org.apache.hadoop.hdfs.DFSClient.getFileChecksumInternal(DFSClient.java:1851)
at
org.apache.hadoop.hdfs.DFSClient.getFileChecksumWithCombineMode(DFSClient.java:1871)
at
org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1891)
at
org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1888)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1905)
at
org.apache.hadoop.hdfs.TestFileChecksum.getFileChecksum(TestFileChecksum.java:584)
at
org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery(TestFileChecksum.java:295)
at
org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery1(TestFileChecksum.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 488892)
Time Spent: 2.5h (was: 2h 20m)
> Add OutputStream + Syncable to the Filesystem Specification
> ---
>
> Key: HADOOP-13327
> URL: https://issues.apache.org/jira/browse/HADOOP-13327
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter:
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=44&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-44 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 23/Sep/20 04:16 Start Date: 23/Sep/20 04:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696394844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 44) Time Spent: 2h 20m (was: 2h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=488298&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488298
]
ASF GitHub Bot logged work on HADOOP-13327:
---
Author: ASF GitHub Bot
Created on: 22/Sep/20 14:26
Start Date: 22/Sep/20 14:26
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2102:
URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696757197
TestRawLocalContract create is failing until we do our own
BufferedOutputStream in passthrough of HADOOP-16830/ #2323 .
the HDFS checksum ones are about checksums not being found, which implies
they weren't being written. Which goes near output streams, doesn't it? Which
means there is a risk this is a genuine regression and not "hdfs tests being
flaky"
```
org.apache.hadoop.fs.PathIOException: `/striped/stripedFileChecksum1': Fail
to get block checksum for
LocatedStripedBlock{BP-1893408133-172.17.0.3-1600724641689:blk_-9223372036854775792_1001;
getBlockSize()=37748736; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:33131,DS-5c8811e5-39af-4967-b3a7-3b76f95b0317,DISK],
DatanodeInfoWithStorage[127.0.0.1:38603,DS-82754ee5-286a-402b-9861-b3ebbc149849,DISK],
DatanodeInfoWithStorage[127.0.0.1:44379,DS-080d6ce9-cb6a-4f80-b37a-e63ecc31d9bc,DISK],
DatanodeInfoWithStorage[127.0.0.1:34723,DS-c6c2aa4e-639b-4d85-9564-05631c8c5b79,DISK],
DatanodeInfoWithStorage[127.0.0.1:40061,DS-54eb3c16-b9ce-4a5d-a7a3-f33b635579b0,DISK],
DatanodeInfoWithStorage[127.0.0.1:45525,DS-3c9ac850-9273-4d2d-933c-2ac3b4b30308,DISK],
DatanodeInfoWithStorage[127.0.0.1:45537,DS-9b1d805c-3b0e-4c84-ad0e-f454817b6829,DISK],
DatanodeInfoWithStorage[127.0.0.1:44697,DS-e06897c8-c8a5-41d6-b5bc-57c7339bbf9b,DISK]];
indices=[1, 2, 3, 4, 5, 6, 7, 8]}
at
org.apache.hadoop.hdfs.FileChecksumHelper$StripedFileNonStripedChecksumComputer.checksumBlocks(FileChecksumHelper.java:640)
at
org.apache.hadoop.hdfs.FileChecksumHelper$FileChecksumComputer.compute(FileChecksumHelper.java:252)
at
org.apache.hadoop.hdfs.DFSClient.getFileChecksumInternal(DFSClient.java:1851)
at
org.apache.hadoop.hdfs.DFSClient.getFileChecksumWithCombineMode(DFSClient.java:1871)
at
org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1891)
at
org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1888)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1905)
at
org.apache.hadoop.hdfs.TestFileChecksum.getFileChecksum(TestFileChecksum.java:584)
at
org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery(TestFileChecksum.java:295)
at
org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery1(TestFileChecksum.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 488298)
Time Spent: 2h 10m (was: 2h)
> Add OutputStream + Syncable to the Filesystem Specification
> ---
>
> Key: HADOOP-13327
> URL: https://issues.apache.org/jira/browse/HADOOP-13327
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: S
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=488293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488293 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 14:21 Start Date: 22/Sep/20 14:21 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696394844 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 11s | trunk passed | | +1 :green_heart: | compile | 19m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 3s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 43s | trunk passed | | +1 :green_heart: | mvnsite | 4m 34s | trunk passed | | +1 :green_heart: | shadedclient | 22m 3s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 1s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 20s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 53s | the patch passed | | +1 :green_heart: | compile | 18m 45s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 45s | the patch passed | | +1 :green_heart: | compile | 16m 46s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 46s | the patch passed | | -0 :warning: | checkstyle | 2m 44s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 29s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 17s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 2s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 36s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 53s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 27s | hadoop-common in the patch passed. | | -1 :x: | unit | 95m 26s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 55s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 21s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. | | | | 294m 41s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestStripedFileAppend | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-had
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=488292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488292 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 14:21 Start Date: 22/Sep/20 14:21 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696401829 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 12s | trunk passed | | +1 :green_heart: | compile | 19m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 16m 52s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 47s | trunk passed | | +1 :green_heart: | mvnsite | 4m 29s | trunk passed | | +1 :green_heart: | shadedclient | 22m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 3s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 18s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 18m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 54s | the patch passed | | +1 :green_heart: | compile | 16m 52s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 52s | the patch passed | | -0 :warning: | checkstyle | 2m 41s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 29s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 0s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 56s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 25s | hadoop-common in the patch passed. | | -1 :x: | unit | 96m 2s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 47s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 20s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | | | | 322m 30s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestDecommissionWithStriped | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | Subsystem | Report/Notes |
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487589 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 03:18 Start Date: 22/Sep/20 03:18 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r492198720 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md ## @@ -629,10 +633,18 @@ The result is `FSDataOutputStream`, which through its operations may generate ne clients creating files with `overwrite==true` to fail if the file is created by another client between the two tests. -* S3A, Swift and potentially other Object Stores do not currently change the FS state +* S3A, Swift and potentially other Object Stores do not currently change the `FS` state until the output stream `close()` operation is completed. -This MAY be a bug, as it allows >1 client to create a file with `overwrite==false`, - and potentially confuse file/directory logic +This is a significant difference between the behavior of object stores +and that of filesystems, as it allows >1 client to create a file with `overwrite==false`, Review comment: changed to ==; also replace >1 with > to keep everything happy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487589) Time Spent: 1.5h (was: 1h 20m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487516 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 03:12 Start Date: 22/Sep/20 03:12 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696394844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487516) Time Spent: 1h 10m (was: 1h) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487559 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 03:16 Start Date: 22/Sep/20 03:16 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696236937 @joshelser -rebased and addressed your minor points I think you are right, we don't need separate HSYNC/HFLUSH options, it's just that they've been there (separately) for a while. What to do? * maintain as is * add SYNCABLE which says "both" * Declare that HSYNC is sufficient and the only one you should look for. If you implement HFLUSH and not HSYNC, you are of no use to applications. retain support for HFLUSH probe, but in the spec only mention it as of historical interest. I like option 3, now I think about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487559) Time Spent: 1h 20m (was: 1h 10m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487593 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 22/Sep/20 03:19 Start Date: 22/Sep/20 03:19 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-650435353 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 53s | trunk passed | | +1 :green_heart: | compile | 21m 31s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 | | +1 :green_heart: | compile | 17m 37s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | checkstyle | 2m 51s | trunk passed | | +1 :green_heart: | mvnsite | 4m 5s | trunk passed | | +1 :green_heart: | shadedclient | 22m 59s | branch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 35s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 39s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 31s | hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 28s | hadoop-azure-datalake in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | +1 :green_heart: | javadoc | 2m 39s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 23s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 22m 5s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 | | +1 :green_heart: | javac | 22m 5s | the patch passed | | +1 :green_heart: | compile | 18m 34s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | javac | 18m 34s | the patch passed | | -0 :warning: | checkstyle | 2m 57s | root: The patch generated 2 new + 105 unchanged - 4 fixed = 107 total (was 109) | | +1 :green_heart: | mvnsite | 3m 55s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 31s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 34s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 38s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 30s | hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 27s | hadoop-azure-datalake in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | +1 :green_heart: | javadoc | 2m 35s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | findbugs | 7m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 30s | hadoop-common in the patch passed. | | -1 :x: | unit | 92m 4s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 30s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 4s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 285m 26s | | | Reason | Tests
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487303&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487303 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 22:02 Start Date: 21/Sep/20 22:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696401829 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 12s | trunk passed | | +1 :green_heart: | compile | 19m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 16m 52s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 47s | trunk passed | | +1 :green_heart: | mvnsite | 4m 29s | trunk passed | | +1 :green_heart: | shadedclient | 22m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 3s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 18s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 18m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 54s | the patch passed | | +1 :green_heart: | compile | 16m 52s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 52s | the patch passed | | -0 :warning: | checkstyle | 2m 41s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 29s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 0s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 56s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 25s | hadoop-common in the patch passed. | | -1 :x: | unit | 96m 2s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 47s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 20s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | | | | 322m 30s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestDecommissionWithStriped | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | Subsystem | Report/Notes | |---
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487304&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487304 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 22:02 Start Date: 21/Sep/20 22:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696402075 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 9s | trunk passed | | +1 :green_heart: | compile | 23m 14s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 19m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 47s | trunk passed | | +1 :green_heart: | mvnsite | 4m 40s | trunk passed | | +1 :green_heart: | shadedclient | 23m 32s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 5s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 43s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 21m 10s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 21m 10s | the patch passed | | +1 :green_heart: | compile | 18m 10s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 18m 10s | the patch passed | | -0 :warning: | checkstyle | 2m 47s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 57s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 6s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 44s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 21s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 1s | hadoop-common in the patch passed. | | -1 :x: | unit | 101m 4s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 58s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 19s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 316m 36s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSnapshotCommands | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2102/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2102 | | Optional Tests
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487295&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487295 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 21:45 Start Date: 21/Sep/20 21:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696394844 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 11s | trunk passed | | +1 :green_heart: | compile | 19m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 3s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 43s | trunk passed | | +1 :green_heart: | mvnsite | 4m 34s | trunk passed | | +1 :green_heart: | shadedclient | 22m 3s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 1s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 20s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 53s | the patch passed | | +1 :green_heart: | compile | 18m 45s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 45s | the patch passed | | +1 :green_heart: | compile | 16m 46s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 46s | the patch passed | | -0 :warning: | checkstyle | 2m 44s | root: The patch generated 3 new + 105 unchanged - 4 fixed = 108 total (was 109) | | +1 :green_heart: | mvnsite | 4m 29s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 17s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 2s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 36s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 53s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 27s | hadoop-common in the patch passed. | | -1 :x: | unit | 95m 26s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 55s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 21s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. | | | | 294m 41s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractCreate | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestStripedFileAppend | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apac
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487071 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 16:48 Start Date: 21/Sep/20 16:48 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-696236937 @joshelser -rebased and addressed your minor points I think you are right, we don't need separate HSYNC/HFLUSH options, it's just that they've been there (separately) for a while. What to do? * maintain as is * add SYNCABLE which says "both" * Declare that HSYNC is sufficient and the only one you should look for. If you implement HFLUSH and not HSYNC, you are of no use to applications. retain support for HFLUSH probe, but in the spec only mention it as of historical interest. I like option 3, now I think about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487071) Time Spent: 0.5h (was: 20m) > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487053 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 16:38 Start Date: 21/Sep/20 16:38 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#discussion_r492198720 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md ## @@ -629,10 +633,18 @@ The result is `FSDataOutputStream`, which through its operations may generate ne clients creating files with `overwrite==true` to fail if the file is created by another client between the two tests. -* S3A, Swift and potentially other Object Stores do not currently change the FS state +* S3A, Swift and potentially other Object Stores do not currently change the `FS` state until the output stream `close()` operation is completed. -This MAY be a bug, as it allows >1 client to create a file with `overwrite==false`, - and potentially confuse file/directory logic +This is a significant difference between the behavior of object stores +and that of filesystems, as it allows >1 client to create a file with `overwrite==false`, Review comment: changed to ==; also replace >1 with > to keep everything happy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 487053) Remaining Estimate: 0h Time Spent: 10m > Add OutputStream + Syncable to the Filesystem Specification > --- > > Key: HADOOP-13327 > URL: https://issues.apache.org/jira/browse/HADOOP-13327 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-13327-002.patch, HADOOP-13327-003.patch, > HADOOP-13327-branch-2-001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Write down what a Filesystem output stream should do. While core the API is > defined in Java, that doesn't say what's expected about visibility, > durability, etc —and Hadoop Syncable interface is entirely ours to define. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[ https://issues.apache.org/jira/browse/HADOOP-13327?focusedWorklogId=487054&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487054 ] ASF GitHub Bot logged work on HADOOP-13327: --- Author: ASF GitHub Bot Created on: 21/Sep/20 16:38 Start Date: 21/Sep/20 16:38 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2102: URL: https://github.com/apache/hadoop/pull/2102#issuecomment-650435353 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 53s | trunk passed | | +1 :green_heart: | compile | 21m 31s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 | | +1 :green_heart: | compile | 17m 37s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | checkstyle | 2m 51s | trunk passed | | +1 :green_heart: | mvnsite | 4m 5s | trunk passed | | +1 :green_heart: | shadedclient | 22m 59s | branch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 35s | hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 39s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 31s | hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 28s | hadoop-azure-datalake in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | +1 :green_heart: | javadoc | 2m 39s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 23s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 22m 5s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 | | +1 :green_heart: | javac | 22m 5s | the patch passed | | +1 :green_heart: | compile | 18m 34s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | javac | 18m 34s | the patch passed | | -0 :warning: | checkstyle | 2m 57s | root: The patch generated 2 new + 105 unchanged - 4 fixed = 107 total (was 109) | | +1 :green_heart: | mvnsite | 3m 55s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 31s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 34s | hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 38s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 30s | hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | -1 :x: | javadoc | 0m 27s | hadoop-azure-datalake in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. | | +1 :green_heart: | javadoc | 2m 35s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | +1 :green_heart: | findbugs | 7m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 30s | hadoop-common in the patch passed. | | -1 :x: | unit | 92m 4s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 1m 30s | hadoop-azure in the patch passed. | | +1 :green_heart: | unit | 1m 4s | hadoop-azure-datalake in the patch passed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 285m 26s | | | Reason | Tests
