subject:"\[jira\] \[Work logged\] \(HADOOP\-16202\) Stabilize openFile\(\) and adopt internally"

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=753156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-753156
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 06/Apr/22 00:29
Start Date: 06/Apr/22 00:29
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on PR #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1089573035

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 58s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  16m  0s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  28m 22s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  25m 11s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  21m 34s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   4m  4s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 54s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 29s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 53s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m  0s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 17s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 22s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 30s | 
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | -1 :x: |  compile  |  23m  3s | 
[/patch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  javac  |  23m  3s | 
[/patch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-compile-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  compile  |  20m 29s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.  |
   | -1 :x: |  javac  |  20m 29s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   3m 53s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 6 new + 838 unchanged - 2 fixed = 844 total (was 
840)  |
   | -1 :x: |  mvnsite  |   0m 44s | 
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | +1 :green_heart: |  javadoc  |   5m 20s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javadoc  |   0m 38s | 
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/19/ar

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=752186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-752186
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 04/Apr/22 10:44
Start Date: 04/Apr/22 10:44
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on PR #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1087397803

   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:29:import
 java.util.concurrent.CompletableFuture;:8: Unused import - 
java.util.concurrent.CompletableFuture. [UnusedImports]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:35:import
 org.apache.hadoop.fs.impl.AbstractFSBuilderImpl;:8: Unused import - 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl. [UnusedImports]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:37:import
 org.apache.hadoop.fs.impl.OpenFileParameters;:8: Unused import - 
org.apache.hadoop.fs.impl.OpenFileParameters. [UnusedImports]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:44:import
 org.apache.hadoop.util.LambdaUtils;:8: Unused import - 
org.apache.hadoop.util.LambdaUtils. [UnusedImports]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:47:import
 static 
org.apache.hadoop.fs.Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS;:15:
 Unused import - 
org.apache.hadoop.fs.Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS.
 [UnusedImports]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:528:
  /**: First sentence should end with a period. [JavadocStyle]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:618:
  public static int DEFAULT_ASYNC_DRAIN_THRESHOLD = 16_000;:21: Name 
'DEFAULT_ASYNC_DRAIN_THRESHOLD' must match pattern '^[a-z][a-zA-Z0-9]*$'. 
[StaticVariableName]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:618:
  public static int DEFAULT_ASYNC_DRAIN_THRESHOLD = 16_000;:21: Variable 
'DEFAULT_ASYNC_DRAIN_THRESHOLD' must be private and have accessor methods. 
[VisibilityModifier]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:117:
@Retries.OnceTranslated: 'method def modifier' has incorrect indentation 
level 4, expected level should be 2. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:121:
  try (DurationInfo ignored = new DurationInfo(LOG, false, "%s", action)) 
{: 'try' has incorrect indentation level 6, expected level should be 4. 
[Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:122:
return operation.apply();: 'try' child has incorrect indentation level 
8, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:123:
  } catch (AmazonClientException e) {: 'try rcurly' has incorrect 
indentation level 6, expected level should be 4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:124:
throw S3AUtils.translateException(action, path, e);: 'catch' child has 
incorrect indentation level 8, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:125:
  }: 'catch rcurly' has incorrect indentation level 6, expected level 
should be 4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:126:
}: 'method def rcurly' has incorrect indentation level 4, expected level 
should be 2. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:137:
@Retries.OnceTranslated: 'method def modifier' has incorrect indentation 
level 4, expected level should be 2. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:144:
  try {: 'try' has incorrect indentation level 6, expected level should be 
4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:145:
return invokeTrackingDuration(tracker, operation);: 'try' child has 
incorrect indentation level 8, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:146:
  } catch (AmazonClientException e) {: 'try rcurly' has incorrect 
indentation level 6, expected level should be 4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java:147:
throw S3AUtil

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=751809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751809
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Apr/22 21:41
Start Date: 01/Apr/22 21:41
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1086351020


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  1s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 49s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m 18s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  24m 59s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  21m 19s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 55s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 56s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 24s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 48s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  10m 46s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 11s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  24m 21s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  24m 21s | 
[/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/18/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1810 unchanged - 0 
fixed = 1812 total (was 1810)  |
   | +1 :green_heart: |  compile  |  21m 16s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  21m 16s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/18/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1684 
unchanged - 0 fixed = 1686 total (was 1684)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   3m 53s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/18/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 25 new + 838 unchanged - 2 fixed = 863 total (was 
840)  |
   | +1 :green_heart: |  mvnsite  |   6m 47s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   5m 21s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javadoc  |   0m 38s | 
[/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/18/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  hadoop-aws in the patch failed with JDK Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.  |
   | -1 :x: |  spotbugs  |   1m 26s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/18/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  24m 30s |  |  patch has no erro

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=751787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751787
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Apr/22 20:20
Start Date: 01/Apr/22 20:20
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1086291097


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 40s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 55s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 14s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  20m 17s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 48s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 59s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 38s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 30s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 18s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 27s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m  6s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  22m  6s | 
[/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/17/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1810 unchanged - 0 
fixed = 1812 total (was 1810)  |
   | +1 :green_heart: |  compile  |  19m 59s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 59s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/17/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1684 
unchanged - 0 fixed = 1686 total (was 1684)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   3m 43s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/17/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 20 new + 838 unchanged - 2 fixed = 858 total (was 
840)  |
   | +1 :green_heart: |  mvnsite  |   7m 56s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javadoc  |   0m 48s | 
[/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/17/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  hadoop-aws in the patch failed with JDK Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.  |
   | -1 :x: |  spotbugs  |   1m 32s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/17/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  21m 38s |  |  patch has no erro

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=751636&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751636
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Apr/22 15:47
Start Date: 01/Apr/22 15:47
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r840714093



##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,522 @@
+
+
+# `FileSystem.openFile()`/`FileContext.openFile()`
+
+This is a method provided by both FileSystem and FileContext for
+advanced file opening options and, where implemented, 
+an asynchrounous/lazy opening of a file.
+
+Creates a builder to open a file, supporting options
+both standard and filesystem specific. The return
+value of the `build()` call is a `Future`,
+which must be waited on. The file opening may be
+asynchronous, and it may actually be postponed (including
+permission/existence checks) until reads are actually
+performed.
+
+This API call was added to `FileSystem` and `FileContext` in
+Hadoop 3.3.0; it was tuned in Hadoop 3.3.1 as follows.
+
+* Added `opt(key, long)` and `must(key, long)`. 
+* Declared that `withFileStatus(null)` is allowed.
+* Declared that `withFileStatus(status)` only checks
+  the filename of the path, not the full path.
+  This is needed to support passthrough/mounted filesystems.
+* Added standard option keys.
+
+###   `FutureDataInputStreamBuilder openFile(Path 
path)`
+
+Creates a [`FutureDataInputStreamBuilder`](fsdatainputstreambuilder.html)
+to construct a operation to open the file at `path` for reading.
+
+When `build()` is invoked on the returned `FutureDataInputStreamBuilder` 
instance,
+the builder parameters are verified and
+`FileSystem.openFileWithOptions(Path, OpenFileParameters)` or
+`AbstractFileSystem.openFileWithOptions(Path, OpenFileParameters)` invoked.
+
+These protected methods returns a `CompletableFuture`
+which, when its `get()` method is called, either returns an input
+stream of the contents of opened file, or raises an exception.
+
+The base implementation of the `FileSystem.openFileWithOptions(PathHandle, 
OpenFileParameters)`
+ultimately invokes `FileSystem.open(Path, int)`.
+
+Thus the chain `FileSystem.openFile(path).build().get()` has the same 
preconditions
+and postconditions as `FileSystem.open(Path p, int bufferSize)`
+
+However, there is one difference which implementations are free to
+take advantage of: 
+
+The returned stream MAY implement a lazy open where file non-existence or
+access permission failures may not surface until the first `read()` of the
+actual data.
+
+The `openFile()` operation MAY check the state of the filesystem during its
+invocation, but as the state of the filesystem may change betwen this call and
+the actual `build()` and `get()` operations, this file-specific
+preconditions (file exists, file is readable, etc) MUST NOT be checked here.
+
+FileSystem implementations which do not implement `open(Path, int)`
+MAY postpone raising an `UnsupportedOperationException` until either the
+`FutureDataInputStreamBuilder.build()` or the subsequent `get()` call,
+else they MAY fail fast in the `openFile()` call.
+
+### Implementors notes
+
+The base implementation of `openFileWithOptions()` actually executes
+the `open(path)` operation synchronously, yet still returns the result
+or any failures in the `CompletableFuture<>`, so as to ensure that users
+code expecting this.
+
+Any filesystem where the time to open a file may be significant SHOULD
+execute it asynchronously by submitting the operation in some executor/thread
+pool. This is particularly recommended for object stores and other filesystems
+likely to be accessed over long-haul connections.
+
+Arbitrary filesystem-specific options MAY be supported; these MUST
+be prefixed with either the filesystem schema, e.g. `hdfs.`
+or in the `fs.SCHEMA` format as normal configuration settings `fs.hdfs`. The
+latter style allows the same configuration option to be used for both
+filesystem configuration and file-specific configuration.
+
+It SHOULD be possible to always open a file without specifying any options,
+so as to present a consistent model to users. However, an implementation MAY
+opt to require one or more mandatory options to be set.
+
+The returned stream may perform "lazy" evaluation of file access. This is
+relevant for object stores where the probes for existence are expensive, and,
+even with an asynchronous open, may be considered needless.
+ 
+###  `FutureDataInputStreamBuilder 
openFile(PathHandle)`
+
+Creates a [`FutureDataInputStreamBuilder`](fsdatainputstreambuilder.html)
+to construct a operation to open the file identified by the given `PathHandle` 
for

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=751631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751631
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Apr/22 15:18
Start Date: 01/Apr/22 15:18
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1086028402


   * rebase against trunk
   * s3a input stream will drain the inner stream asynchronously in 
seek/unbuffer related calls if the # of bytes to drain is > a new 
config/openFile option `fs.s3a.input.async.drain.threshold`; default is 16000 
bytes, which seems a good number in my long haul experiments. also draining is 
into a 16k byte buffer, which speeds it up.
   
   This aims to reduce the cost of seeking where bytes do need to be discarded.
   
   the time to drain is also measured. it's the max time which can be high, as 
it is time to read bytes remaining in the current read. abort is less expensive 
in the actual abort...its the negotiation of a new TLS channel later which 
costs.
   
   ```
   (stream_read_remote_stream_aborted.mean=(samples=2, sum=2, mean=1.))
   (stream_read_remote_stream_drain.mean=(samples=13, sum=29, mean=2.2308)));
   
   (stream_read_remote_stream_aborted.max=1)
   (stream_read_remote_stream_drain.max=25));
   ```
   
   the `ITestS3AInputStreamPerformance` suite also sets the file length in 
every openFile call. so skips all creation, saves a few seconds overall, 
showing it is tangible.
   
   full stats from a remote testrun
   
   ```
   
   2022-04-01 15:49:10,887 [JUnit] INFO  s3a.AbstractS3ATestBase 
(AbstractS3ATestBase.java:dumpFileSystemIOStatistics(123)) - Aggregate 
FileSystem Statistics counters=((action_executor_acquired=1)
   (action_file_opened=8)
   (action_http_get_request=15)
   (action_http_head_request=25)
   (audit_request_execution=76)
   (audit_span_creation=50)
   (directories_created=8)
   (directories_deleted=7)
   (fake_directories_deleted=1)
   (files_created=1)
   (files_deleted=1)
   (object_bulk_delete_request=2)
   (object_delete_objects=9)
   (object_delete_request=7)
   (object_list_request=18)
   (object_metadata_request=25)
   (object_put_bytes=32768)
   (object_put_request=9)
   (object_put_request_completed=9)
   (op_create=1)
   (op_delete=8)
   (op_get_file_status=9)
   (op_mkdirs=8)
   (op_open=8)
   (store_io_request=78)
   (stream_aborted=2)
   (stream_read_bytes=93473433)
   (stream_read_bytes_backwards_on_seek=12713984)
   (stream_read_bytes_discarded_in_abort=43889622)
   (stream_read_bytes_discarded_in_close=252395)
   (stream_read_close_operations=8)
   (stream_read_closed=13)
   (stream_read_fully_operations=8)
   (stream_read_opened=15)
   (stream_read_operations=6124)
   (stream_read_operations_incomplete=6071)
   (stream_read_remote_stream_aborted=2)
   (stream_read_remote_stream_drain=13)
   (stream_read_seek_backward_operations=4)
   (stream_read_seek_bytes_discarded=45092691)
   (stream_read_seek_bytes_skipped=55054163)
   (stream_read_seek_forward_operations=175)
   (stream_read_seek_operations=179)
   (stream_read_seek_policy_changed=9)
   (stream_read_total_bytes=138818519)
   (stream_write_block_uploads=1)
   (stream_write_bytes=32768)
   (stream_write_total_data=65536));
   
   gauges=((stream_write_block_uploads_pending=1));
   
   minimums=((action_executor_acquired.min=0)
   (action_file_opened.min=0)
   (action_http_get_request.min=31)
   (action_http_head_request.min=21)
   (object_bulk_delete_request.min=37)
   (object_delete_request.min=28)
   (object_list_request.min=27)
   (object_put_request.min=60)
   (op_create.min=61)
   (op_delete.min=28)
   (op_get_file_status.min=35)
   (op_mkdirs.min=155)
   (stream_read_remote_stream_aborted.min=1)
   (stream_read_remote_stream_drain.min=0));
   
   maximums=((action_executor_acquired.max=0)
   (action_file_opened.max=0)
   (action_http_get_request.max=730)
   (action_http_head_request.max=1663)
   (object_bulk_delete_request.max=84)
   (object_delete_request.max=35)
   (object_list_request.max=648)
   (object_put_request.max=205)
   (op_create.max=61)
   (op_delete.max=159)
   (op_get_file_status.max=1669)
   (op_mkdirs.max=769)
   (stream_read_remote_stream_aborted.max=1)
   (stream_read_remote_stream_drain.max=25));
   
   means=((action_executor_acquired.mean=(samples=1, sum=0, mean=0.))
   (action_file_opened.mean=(samples=8, sum=0, mean=0.))
   (action_http_get_request.mean=(samples=15, sum=2752, mean=183.4667))
   (action_http_head_request.mean=(samples=25, sum=7360, mean=294.4000))
   (object_bulk_delete_request.mean=(samples=2, sum=121, mean=60.5000))
   (object_delete_request.mean=(samples=7, sum=213, mean=30.4286))
   (object_list_request.mean=(samples=18, sum=1520, mean=84.))
   (object_put_

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-04-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=751628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751628
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Apr/22 15:08
Start Date: 01/Apr/22 15:08
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1065987294


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 44s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 19s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 15s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 37s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  19m 57s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 44s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 55s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  1s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 10s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 29s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m  1s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  22m  1s | 
[/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1811 unchanged - 0 
fixed = 1813 total (was 1811)  |
   | +1 :green_heart: |  compile  |  19m 47s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 47s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1685 
unchanged - 0 fixed = 1687 total (was 1685)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   3m 44s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 2 new + 836 unchanged - 1 fixed = 838 total (was 
837)  |
   | +1 :green_heart: |  mvnsite  |   7m 55s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 30s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  12m 57s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m  1s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 52s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m  0s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m 45s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 38s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  21m

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-03-22 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745805
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 22/Mar/22 12:53
Start Date: 22/Mar/22 12:53
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1075138032


   checkstyle
   
   ```
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:120:import
 org.apache.hadoop.fs.s3a.select.InternalSelectConstants;:8: Unused import - 
org.apache.hadoop.fs.s3a.select.InternalSelectConstants. [UnusedImports]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:194:import
 static 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: 
Unused import - 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. 
[UnusedImports]'
   ```
   
   javac warnings
   ```
   
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4099:14:[deprecation]
 getDefaultBlockSize() in FileSystem has been deprecated
   
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked]
 unchecked method invocation: method sort in interface List is applied to given 
types
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 745805)
Time Spent: 15h  (was: 14h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 15h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-03-22 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745799
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 22/Mar/22 12:52
Start Date: 22/Mar/22 12:52
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 745799)
Time Spent: 14h 50m  (was: 14h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 14h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-03-12 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=740485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-740485
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 12/Mar/22 23:59
Start Date: 12/Mar/22 23:59
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1065987294


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 44s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 19s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 15s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 37s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  19m 57s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 44s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 55s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  1s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 10s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 29s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m  1s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  22m  1s | 
[/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1811 unchanged - 0 
fixed = 1813 total (was 1811)  |
   | +1 :green_heart: |  compile  |  19m 47s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 47s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1685 
unchanged - 0 fixed = 1687 total (was 1685)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   3m 44s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/16/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 2 new + 836 unchanged - 1 fixed = 838 total (was 
837)  |
   | +1 :green_heart: |  mvnsite  |   7m 55s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 30s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  12m 57s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m  1s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 52s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m  0s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m 45s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 38s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  21m  0s |  |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-02-18 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=729664&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729664
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 18/Feb/22 13:48
Start Date: 18/Feb/22 13:48
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1044553287


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 45s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 41s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m 15s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 59s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  20m 54s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 52s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 43s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 44s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 34s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 35s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 59s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 49s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m 40s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  22m 40s | 
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/15/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 2 new + 1811 unchanged - 0 
fixed = 1813 total (was 1811)  |
   | +1 :green_heart: |  compile  |  21m 56s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  21m 56s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/15/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1685 
unchanged - 0 fixed = 1687 total (was 1685)  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   3m 47s |  |  root: The patch generated 
0 new + 836 unchanged - 1 fixed = 836 total (was 837)  |
   | +1 :green_heart: |  mvnsite  |   7m 51s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   5m 50s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 37s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  13m 22s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m 14s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  25m 10s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m  1s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m 42s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 36s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  19m 44s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  4s |  |  hadoop-mapreduce-examples in 
the patch passed.

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=729228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729228
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Feb/22 20:32
Start Date: 17/Feb/22 20:32
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r809443998



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
##
@@ -390,7 +395,32 @@ public static boolean copy(FileSystem srcFS, Path src,
 return copy(srcFS, fileStatus, dstFS, dst, deleteSource, overwrite, conf);
   }
 
-  /** Copy files between FileSystems. */
+  /**
+   * Copy a file/directory tree within/between filesystems.
+   * 
+   * returns true if the operation succeeded. When deleteSource is true,
+   * this means "after the copy, delete(source) returned true"
+   * If the destination is a directory, and mkdirs (dest) fails,
+   * the operation will return false rather than raise any exception.
+   * 
+   * The overwrite flag is about overwriting files; it has no effect about
+   * handing an attempt to copy a file atop a directory (expect an 
IOException),
+   * or a directory over a path which contains a file (mkdir will fail, so
+   * "false").
+   * 
+   * The operation is recursive, and the deleteSource operation takes place
+   * as each subdirectory is copied. Therefore, if an operation fails partway
+   * through, the source tree may be partially deleted.
+   * @param srcFS source filesystem
+   * @param srcStatus status of source
+   * @param dstFS destination filesystem
+   * @param dst path of source
+   * @param deleteSource delete the source?
+   * @param overwrite overwrite files at destination?
+   * @param conf configuration to use when opening files
+   * @return true if the operation succeeded.

Review comment:
   none. its just always been that way. and yes, it is wrong




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729228)
Time Spent: 14h 20m  (was: 14h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 14h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717940
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 31/Jan/22 14:32
Start Date: 31/Jan/22 14:32
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 45s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  11m 33s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  27m  9s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  27m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   5m 10s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 55s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 45s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   2m  1s | 
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
 |  hadoop-yarn-common in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 59s | 
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt)
 |  hadoop-mapreduce-client-app in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 34s | 
[/branch-spotbugs-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-tools_hadoop-distcp.txt)
 |  hadoop-distcp in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 25s | 
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt)
 |  hadoop-mapreduce-examples in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 42s | 
[/branch-spotbugs-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-tools_hadoop-streaming.txt)
 |  hadoop-streaming in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 40s | 
[/branch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in trunk failed.  |
   | +1 :green_heart: |  shadedclient  |   2m 59s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 20s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 24s | 
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 22s | 
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt)
 |  hadoop-mapreduce-client-app in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 23s | 
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/14/artifact/out/patch-mvninstall-hadoop-ma

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717345
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 22:47
Start Date: 28/Jan/22 22:47
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1024721403


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  9s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  11m 32s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  24m 28s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  25m 11s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  20m 59s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 58s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 52s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 38s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 54s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 48s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 15s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 30s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m  1s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  24m  4s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  24m  4s | 
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/13/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 2 new + 1811 unchanged - 0 
fixed = 1813 total (was 1811)  |
   | +1 :green_heart: |  compile  |  21m 10s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  21m 10s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/13/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1685 
unchanged - 0 fixed = 1687 total (was 1685)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/13/artifact/out/blanks-eol.txt)
 |  The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 55s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/13/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 2 new + 836 unchanged - 1 fixed = 838 total (was 
837)  |
   | +1 :green_heart: |  mvnsite  |   8m 12s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 25s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 52s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   1m 38s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/13/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  23m 27s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :gr

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717202
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 18:37
Start Date: 28/Jan/22 18:37
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1024502247


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  11m 38s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m 29s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 44s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 59s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 35s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 51s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 43s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 37s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  21m 37s | 
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/12/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 2 new + 1811 unchanged - 0 
fixed = 1813 total (was 1811)  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 33s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/12/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 1685 
unchanged - 0 fixed = 1687 total (was 1685)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/12/artifact/out/blanks-eol.txt)
 |  The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 43s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/12/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 1 new + 818 unchanged - 1 fixed = 819 total (was 
819)  |
   | +1 :green_heart: |  mvnsite  |   7m 56s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   1m 32s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/12/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  21m 21s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :gr

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717128
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 16:31
Start Date: 28/Jan/22 16:31
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1015830318


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 43s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 53s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m 36s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 13s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 45s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 53s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 33s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  6s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 25s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 59s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 41s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 40s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  21m 40s | 
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 7 new + 1811 unchanged - 0 
fixed = 1818 total (was 1811)  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 33s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 7 new + 1685 
unchanged - 0 fixed = 1692 total (was 1685)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/blanks-eol.txt)
 |  The patch has 12 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 43s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 11 new + 818 unchanged - 1 fixed = 829 total (was 
819)  |
   | +1 :green_heart: |  mvnsite  |   7m 56s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 35s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   1m 32s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  21m 23s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717022&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717022
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 12:35
Start Date: 28/Jan/22 12:35
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-914219063






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 717022)
Time Spent: 13.5h  (was: 13h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717021
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 12:34
Start Date: 28/Jan/22 12:34
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-907475705


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 41s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 52s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 26s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  18m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   3m 53s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 30s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 33s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 25s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  11m  1s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  18m  2s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 31s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 18s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  27m 23s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  27m 23s | 
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1916 unchanged - 0 
fixed = 1917 total (was 1916)  |
   | +1 :green_heart: |  compile  |  24m 13s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  javac  |  24m 13s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1790 
unchanged - 0 fixed = 1791 total (was 1790)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/blanks-eol.txt)
 |  The patch has 13 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m 51s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 8 new + 852 unchanged - 1 fixed = 860 total (was 
853)  |
   | +1 :green_heart: |  mvnsite  |   8m  4s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m  2s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  spotbugs  |   1m 43s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  20m 17s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-28 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=717020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717020
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 28/Jan/22 12:34
Start Date: 28/Jan/22 12:34
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-801558206






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 717020)
Time Spent: 13h 10m  (was: 13h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2022-01-18 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=710880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710880
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 18/Jan/22 21:02
Start Date: 18/Jan/22 21:02
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1015830318


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 43s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 53s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m 36s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 13s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   3m 45s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 53s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 33s |  |  trunk passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  6s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  11m 25s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 59s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 41s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 40s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  21m 40s | 
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 7 new + 1811 unchanged - 0 
fixed = 1818 total (was 1811)  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  javac  |  19m 33s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
 |  root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 7 new + 1685 
unchanged - 0 fixed = 1692 total (was 1685)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/blanks-eol.txt)
 |  The patch has 12 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 43s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 11 new + 818 unchanged - 1 fixed = 829 total (was 
819)  |
   | +1 :green_heart: |  mvnsite  |   7m 56s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 35s |  |  the patch passed with JDK 
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   7m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   1m 32s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/11/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  21m 23s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-12-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=689188&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-689188
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 02/Dec/21 11:13
Start Date: 02/Dec/21 11:13
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r760992854



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,112 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {
+
+private OpenFileOptions() {
+}
+
+/**
+ * Prefix for all standard filesystem options: {@value}.
+ */
+private static final String FILESYSTEM_OPTION = "fs.option.";
+
+/**
+ * Prefix for all openFile options: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE =
+FILESYSTEM_OPTION + "openfile.";
+
+/**
+ * OpenFile option for file length: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_LENGTH =
+FS_OPTION_OPENFILE + "length";
+
+/**
+ * OpenFile option for split start: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_START =
+FS_OPTION_OPENFILE + "split.start";
+
+/**
+ * OpenFile option for split end: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_END =
+FS_OPTION_OPENFILE + "split.end";
+
+/**
+ * OpenFile option for buffer size: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_BUFFER_SIZE =
+FS_OPTION_OPENFILE + "buffer.size";
+
+/**
+ * OpenFile option for read policies: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY =
+FS_OPTION_OPENFILE + "read.policy";
+
+/**
+ * Set of standard options which openFile implementations
+ * MUST recognize, even if they ignore the actual values.
+ */
+public static final Set FS_OPTION_OPENFILE_STANDARD_OPTIONS =
+Collections.unmodifiableSet(Stream.of(
+FS_OPTION_OPENFILE_BUFFER_SIZE,
+FS_OPTION_OPENFILE_READ_POLICY,
+FS_OPTION_OPENFILE_LENGTH,
+FS_OPTION_OPENFILE_SPLIT_START,
+FS_OPTION_OPENFILE_SPLIT_END)
+.collect(Collectors.toSet()));
+
+/**
+ * Read policy for adaptive IO: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_ADAPTIVE =
+"adaptive";
+
+/**
+ * Read policy 'normal' -up to implementation: {@value}.

Review comment:
   👎🏽 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 689188)
Time Spent: 12h 50m  (was: 12h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-ma

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-11-24 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=686121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-686121
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 24/Nov/21 22:40
Start Date: 24/Nov/21 22:40
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-978376044


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 49s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 50s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m  5s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  24m 58s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  21m 10s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   3m 56s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   8m 30s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 56s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m 25s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  12m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  22m  0s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 30s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 12s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  23m 56s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  23m 56s | 
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/10/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1889 unchanged - 0 
fixed = 1890 total (was 1889)  |
   | +1 :green_heart: |  compile  |  21m 11s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  javac  |  21m 11s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/10/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1763 
unchanged - 0 fixed = 1764 total (was 1763)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/10/artifact/out/blanks-eol.txt)
 |  The patch has 12 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 56s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/10/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 9 new + 835 unchanged - 1 fixed = 844 total (was 
836)  |
   | +1 :green_heart: |  mvnsite  |   8m 30s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m 56s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   7m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  spotbugs  |   1m 51s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/10/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  22m  7s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :gr

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652409
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:41
Start Date: 17/Sep/21 17:41
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711240136



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java
##
@@ -38,6 +38,9 @@
*/
   private Set mandatoryKeys;
 
+  /** The optional keys. */
+  private Set optionalKeys;

Review comment:
   its covered in the filesystem docs. essentially
   * mandatory keys: something the FS implementation *must* understand. (e.g. 
s3 select SQL query); 
   * optional is ("if you don't recognise the key, ignore the option)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652409)
Time Spent: 12.5h  (was: 12h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652408
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:39
Start Date: 17/Sep/21 17:39
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711239176



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,112 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {
+
+private OpenFileOptions() {
+}
+
+/**
+ * Prefix for all standard filesystem options: {@value}.
+ */
+public static final String FILESYSTEM_OPTION = "fs.option.";
+
+/**
+ * Prefix for all openFile options: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE =
+FILESYSTEM_OPTION + "openfile.";
+
+/**
+ * OpenFile option for file length: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_LENGTH =
+FS_OPTION_OPENFILE + "length";
+
+/**
+ * OpenFile option for split start: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_START =
+FS_OPTION_OPENFILE + "split.start";
+
+/**
+ * OpenFile option for split end: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_END =
+FS_OPTION_OPENFILE + "split.end";
+
+/**
+ * OpenFile option for buffer size: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_BUFFER_SIZE =
+FS_OPTION_OPENFILE + "buffer.size";
+
+/**
+ * OpenFile option for read policies: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY =
+FS_OPTION_OPENFILE + "read.policy";
+
+/**
+ * Read policy for adaptive IO: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_ADAPTIVE =

Review comment:
   oh, and org.apache.hadoop.mapreduce.lib.input.LineRecordReader will set 
the start/end of splits, so while its doing sequential read, something *really* 
clever could know exactly how far to read




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652408)
Time Spent: 12h 20m  (was: 12h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652405&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652405
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:37
Start Date: 17/Sep/21 17:37
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711237755



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,112 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {
+
+private OpenFileOptions() {
+}
+
+/**
+ * Prefix for all standard filesystem options: {@value}.
+ */
+public static final String FILESYSTEM_OPTION = "fs.option.";
+
+/**
+ * Prefix for all openFile options: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE =
+FILESYSTEM_OPTION + "openfile.";
+
+/**
+ * OpenFile option for file length: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_LENGTH =
+FS_OPTION_OPENFILE + "length";
+
+/**
+ * OpenFile option for split start: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_START =
+FS_OPTION_OPENFILE + "split.start";
+
+/**
+ * OpenFile option for split end: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_END =
+FS_OPTION_OPENFILE + "split.end";
+
+/**
+ * OpenFile option for buffer size: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_BUFFER_SIZE =
+FS_OPTION_OPENFILE + "buffer.size";
+
+/**
+ * OpenFile option for read policies: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY =
+FS_OPTION_OPENFILE + "read.policy";
+
+/**
+ * Read policy for adaptive IO: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_ADAPTIVE =

Review comment:
   good point.
   
   I want us to have some basic names of them so they can be used more broadly, 
and in particular, we can have tools asking for some "whole-file", knowing that 
an FS which recognises them will handle them properly.
   
   FWIW whole-file is one I've realised abfs & s3a can do very efficiently
   * switch to a buffer read strategy of big buffers but fewer readers
   * start to read immediately
   
   in contrast, `sequential` is often used for splits, so lazy seek is needed 
and as it may end earlier, smaller buffers still make sense.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652405)
Time Spent: 12h 10m  (was: 12h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652400
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:30
Start Date: 17/Sep/21 17:30
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711233595



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,112 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {
+
+private OpenFileOptions() {
+}
+
+/**
+ * Prefix for all standard filesystem options: {@value}.
+ */
+public static final String FILESYSTEM_OPTION = "fs.option.";
+
+/**
+ * Prefix for all openFile options: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE =
+FILESYSTEM_OPTION + "openfile.";
+
+/**
+ * OpenFile option for file length: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_LENGTH =
+FS_OPTION_OPENFILE + "length";
+
+/**
+ * OpenFile option for split start: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_START =
+FS_OPTION_OPENFILE + "split.start";
+
+/**
+ * OpenFile option for split end: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_SPLIT_END =
+FS_OPTION_OPENFILE + "split.end";
+
+/**
+ * OpenFile option for buffer size: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_BUFFER_SIZE =
+FS_OPTION_OPENFILE + "buffer.size";
+
+/**
+ * OpenFile option for read policies: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY =
+FS_OPTION_OPENFILE + "read.policy";
+
+/**
+ * Read policy for adaptive IO: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_ADAPTIVE =
+"adaptive";
+
+/**
+ * Read policy for ORC files: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_ORC =
+"orc";
+
+/**
+ * Read policy 'normal' -up to implementation: {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_READ_POLICY_NORMAL =

Review comment:
   done. I'll tweak s3a to map "normal" to default.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652400)
Time Spent: 12h  (was: 11h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652399
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:25
Start Date: 17/Sep/21 17:25
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711230698



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Tail.java
##
@@ -107,16 +109,15 @@ private long dumpFromOffset(PathData item, long offset) 
throws IOException {
 if (offset < 0) {
   offset = Math.max(fileSize + offset, 0);
 }
-
-FSDataInputStream in = item.fs.open(item.path);
-try {
+// Always do sequential reads.
+try (FSDataInputStream in = item.openFile(
+FS_OPTION_OPENFILE_READ_POLICY_NORMAL)) {

Review comment:
   fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652399)
Time Spent: 11h 50m  (was: 11h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652397
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:24
Start Date: 17/Sep/21 17:24
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711229603



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,112 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {
+
+private OpenFileOptions() {
+}
+
+/**
+ * Prefix for all standard filesystem options: {@value}.
+ */
+public static final String FILESYSTEM_OPTION = "fs.option.";

Review comment:
   made it private




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652397)
Time Spent: 11.5h  (was: 11h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=652398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652398
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 17/Sep/21 17:24
Start Date: 17/Sep/21 17:24
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r711230017



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
##
@@ -4655,7 +4656,7 @@ public FutureDataInputStreamBuilder openFile(PathHandle 
pathHandle)
   final OpenFileParameters parameters) throws IOException {
 AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(

Review comment:
   I agree, but we need flexibility of linking, ability of clients to work 
with any FS implementation, etc etc




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 652398)
Time Spent: 11h 40m  (was: 11.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-09-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=647309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647309
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 07/Sep/21 11:19
Start Date: 07/Sep/21 11:19
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-914219063


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m 10s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 32s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  25m  6s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  28m 17s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  23m 40s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   4m  1s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 14s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 24s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 38s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  11m  2s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  18m 39s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 27s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  26m  2s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  26m  2s | 
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/9/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1924 unchanged - 0 
fixed = 1925 total (was 1924)  |
   | +1 :green_heart: |  compile  |  21m  5s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  javac  |  21m  5s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/9/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1797 
unchanged - 0 fixed = 1798 total (was 1797)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/9/artifact/out/blanks-eol.txt)
 |  The patch has 13 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 47s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/9/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 8 new + 853 unchanged - 1 fixed = 861 total (was 
854)  |
   | +1 :green_heart: |  mvnsite  |   6m 41s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   5m 18s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 53s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  spotbugs  |   1m 31s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/9/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  17m 33s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_h

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-08-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=643010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-643010
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 27/Aug/21 21:05
Start Date: 27/Aug/21 21:05
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-907475705


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 41s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 52s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 26s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  18m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   3m 53s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 30s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 33s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 25s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  11m  1s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  18m  2s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 31s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 18s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  27m 23s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javac  |  27m 23s | 
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1916 unchanged - 0 
fixed = 1917 total (was 1916)  |
   | +1 :green_heart: |  compile  |  24m 13s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  javac  |  24m 13s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1790 
unchanged - 0 fixed = 1791 total (was 1790)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/blanks-eol.txt)
 |  The patch has 13 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m 51s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 8 new + 852 unchanged - 1 fixed = 860 total (was 
853)  |
   | +1 :green_heart: |  mvnsite  |   8m  4s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   6m  2s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  spotbugs  |   1m 43s | 
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/8/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
 |  hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total 
(was 0)  |
   | +1 :green_heart: |  shadedclient  |  20m 17s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  u

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-08-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=642927&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642927
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 27/Aug/21 17:27
Start Date: 27/Aug/21 17:27
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-906841571


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 55s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 33s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m  1s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  19m 20s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   3m 59s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 40s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 13s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  10m 29s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m 21s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 16s | 
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | -1 :x: |  compile  |  20m 45s | 
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  javac  |  20m 45s | 
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  compile  |  18m 28s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.  |
   | -1 :x: |  javac  |  18m 28s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 42s | 
[/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/buildtool-patch-checkstyle-root.txt)
 |  The patch fails to run checkstyle in root  |
   | -1 :x: |  mvnsite  |   0m 32s | 
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | +1 :green_heart: |  javadoc  |   5m  4s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javadoc  |   0m 31s |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-08-26 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=642662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642662
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 27/Aug/21 00:57
Start Date: 27/Aug/21 00:57
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-906841571


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 55s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 17 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 33s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m  1s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  19m 20s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   3m 59s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 40s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   5m 13s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |  10m 29s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m 21s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 16s | 
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | -1 :x: |  compile  |  20m 45s | 
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  javac  |  20m 45s | 
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
 |  root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.  |
   | -1 :x: |  compile  |  18m 28s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.  |
   | -1 :x: |  javac  |  18m 28s | 
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
 |  root in the patch failed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   3m 42s | 
[/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/buildtool-patch-checkstyle-root.txt)
 |  The patch fails to run checkstyle in root  |
   | -1 :x: |  mvnsite  |   0m 32s | 
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
 |  hadoop-aws in the patch failed.  |
   | +1 :green_heart: |  javadoc  |   5m  4s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | -1 :x: |  javadoc  |   0m 31s | 
[/patc

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-07-12 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=621590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-621590
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 12/Jul/21 19:55
Start Date: 12/Jul/21 19:55
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-760560973






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 621590)
Time Spent: 10h 40m  (was: 10.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-04-26 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=589293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-589293
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 26/Apr/21 15:19
Start Date: 26/Apr/21 15:19
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-826924421


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 53s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 47s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m 58s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 19s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  18m 59s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   4m  4s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 19s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   4m 45s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 36s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  10m 22s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  4s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 20s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 14s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 37s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | -1 :x: |  javac  |  21m 37s | 
[/results-compile-javac-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/6/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
 |  root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 1933 unchanged - 0 
fixed = 1934 total (was 1933)  |
   | +1 :green_heart: |  compile  |  18m 55s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | -1 :x: |  javac  |  18m 55s | 
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/6/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
 |  root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 1829 
unchanged - 0 fixed = 1830 total (was 1829)  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/6/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m  3s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/6/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 3 new + 820 unchanged - 1 fixed = 823 total (was 
821)  |
   | +1 :green_heart: |  mvnsite  |   6m 16s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   4m 42s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  13m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  17m 46s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  18m 38s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m 20s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m 32s |  |  hadoop-mapred

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-04-25 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=588865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-588865
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 26/Apr/21 06:55
Start Date: 26/Apr/21 06:55
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-826430226


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 14s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m  2s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m  0s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   4m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 17s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   4m 46s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 38s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  10m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  4s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 39s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  21m 39s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |  19m  0s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/5/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m  6s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/5/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 3 new + 820 unchanged - 1 fixed = 823 total (was 
821)  |
   | +1 :green_heart: |  mvnsite  |   6m 15s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   4m 45s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 40s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  11m 53s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  17m  8s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 25s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 42s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  5s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 22s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  37m 59s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  2s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   6m 49s |  |  hadoop-streaming in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   2m 32s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 50s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 314m 16s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-04-25 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=588641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-588641
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 26/Apr/21 00:42
Start Date: 26/Apr/21 00:42
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-826430226


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 14s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m  2s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m  0s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   4m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 17s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   4m 46s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 38s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  10m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  4s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 39s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  21m 39s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |  19m  0s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/5/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m  6s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/5/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 3 new + 820 unchanged - 1 fixed = 823 total (was 
821)  |
   | +1 :green_heart: |  mvnsite  |   6m 15s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   4m 45s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   5m 40s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  11m 53s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  17m  8s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 25s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 42s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  5s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 22s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  37m 59s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  2s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   6m 49s |  |  hadoop-streaming in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   2m 32s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 50s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 314m 16s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-03-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=568043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-568043
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 18/Mar/21 01:59
Start Date: 18/Mar/21 01:59
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-801558206


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 39s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m 19s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m  8s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 58s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  18m  2s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   3m 55s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 24s |  |  trunk passed  |
   | -1 :x: |  javadoc  |   1m  4s | 
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
 |  hadoop-common in trunk failed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.  |
   | +1 :green_heart: |  javadoc  |   6m 43s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  11m  6s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  14m 27s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 25s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  1s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  20m  1s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 59s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |  17m 59s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/blanks-eol.txt)
 |  The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   4m 26s | 
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/results-checkstyle-root.txt)
 |  root: The patch generated 3 new + 821 unchanged - 2 fixed = 824 total (was 
823)  |
   | +1 :green_heart: |  mvnsite  |   7m 29s |  |  the patch passed  |
   | -1 :x: |  javadoc  |   1m  3s | 
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
 |  hadoop-common in the patch failed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.  |
   | +1 :green_heart: |  javadoc  |   6m 40s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |  12m 38s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  14m 39s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |  18m 53s | 
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  unit  |   4m 55s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m 10s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-02-15 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=552586&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-552586
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 15/Feb/21 15:30
Start Date: 15/Feb/21 15:30
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-779295371


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  14m 30s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m  6s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 21s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 19s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m  3s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   3m 59s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m  6s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 48s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 25s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 13s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  10m 52s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 26s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  3s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  20m  2s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m  1s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |  18m  1s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   4m  0s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 5 new + 821 unchanged - 1 fixed = 826 total (was 
822)  |
   | +1 :green_heart: |  mvnsite  |   7m 18s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
 |  The patch has 10 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  13m  8s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 46s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  findbugs  |  11m 55s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 49s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 20s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m 17s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 29s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 16s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   0m 59s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   6m 38s |  |  hadoop-streaming in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   2m 10s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 55s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 295m 59s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-02-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=546404&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-546404
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 03/Feb/21 01:01
Start Date: 03/Feb/21 01:01
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771228069


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 30s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m  7s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 12s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 45s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  18m  6s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  checkstyle  |   4m 11s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 33s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 54s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  10m 53s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 21s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  8s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  20m  8s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  javac  |  18m  3s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   4m  0s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 5 new + 821 unchanged - 1 fixed = 826 total (was 
822)  |
   | +1 :green_heart: |  mvnsite  |   7m 22s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/whitespace-eol.txt)
 |  The patch has 10 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  13m 12s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 48s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 44s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  findbugs  |  11m 59s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 54s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 43s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m 48s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   9m 49s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  15m 45s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  9s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | -1 :x: |  unit  |   6m 46s | 
[/patch-unit-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-unit-hadoop-tools_hadoop-streaming.txt)
 |  hadoop-streaming in the patch passed.  |
   | +1 :green_heart: |  unit  |   2m 34s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   1m  1s |  |  Th

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-02-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=545690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545690
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Feb/21 23:17
Start Date: 01/Feb/21 23:17
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771228069


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 30s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m  7s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 12s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 45s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  18m  6s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  checkstyle  |   4m 11s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 33s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 54s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  10m 53s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 21s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  8s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  20m  8s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  javac  |  18m  3s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   4m  0s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 5 new + 821 unchanged - 1 fixed = 826 total (was 
822)  |
   | +1 :green_heart: |  mvnsite  |   7m 22s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/whitespace-eol.txt)
 |  The patch has 10 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  13m 12s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 48s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   6m 44s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01  |
   | +1 :green_heart: |  findbugs  |  11m 59s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  17m 54s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 43s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m 48s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   9m 49s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  15m 45s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  9s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | -1 :x: |  unit  |   6m 46s | 
[/patch-unit-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-unit-hadoop-tools_hadoop-streaming.txt)
 |  hadoop-streaming in the patch passed.  |
   | +1 :green_heart: |  unit  |   2m 34s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   1m  1s |  |  Th

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-02-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=545505&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545505
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 01/Feb/21 18:40
Start Date: 01/Feb/21 18:40
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771070179


   rebased to fix compile problems; the final patch is the one with changes 
since Thomas's last review
   
   * move all the text on options into the fsdatainputstreambuilder.md file
   * try to define the validation of must/opt in pseudo-python
   * make sure the doc is consistent in saying FNFE and permission issues MUST 
be delayed until future.get() and MAY be delayed until the first actual read.  
This is to give object stores the maximum time for async probes. Example: you 
could initiate a HEAD request in  build() but not block for its completion 
until read()
   
   S3A added some extra tuning of the input stream/openFile logic, so the 
stream no longer gets access to the S3 client, simply callbacks
   
   Testing: in progress



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 545505)
Time Spent: 9h 20m  (was: 9h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-22 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=540138&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-540138
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 22/Jan/21 11:41
Start Date: 22/Jan/21 11:41
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r562575365



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSBuilder.java
##
@@ -61,6 +61,13 @@
*/
   B opt(@Nonnull String key, float value);
 
+  /**
+   * Set optional long parameter for the Builder.
+   *
+   * @see #opt(String, String)
+   */
+  B opt(@Nonnull String key, long value);
+

Review comment:
   I agree, but as well as it already being out there, I want to let 
applications compile against any version of hadoop with the API, even if a 
specific FS option isn't available, alongside allowing for custom FS opts. As 
an example, I have a PoC of parquet lib which uses this which is designed to 
compile against 3.3.x. (that isn't something I've stuck up as I use it to see 
how this stuff could be added to a library...highlights what is broken right 
now, specifically; S3A openFile.withFileStatus fails if used via Hive because 
hive wraps the FileStatus to a different type from S3AFileStatus. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 540138)
Time Spent: 9h 10m  (was: 9h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-21 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=539749&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-539749
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 22/Jan/21 06:09
Start Date: 22/Jan/21 06:09
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-763994374







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 539749)
Time Spent: 9h  (was: 8h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-21 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=538993&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-538993
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 21/Jan/21 11:43
Start Date: 21/Jan/21 11:43
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-764587055


   Thomas, to clarify a bit more
   
   yes, an FS can choose to ignore an option, -but it must recognise the option 
and so make a conscious decision "this doesn't matter"
   
   * so with the standard length/read policy options, an FS can concude "I see 
that and don't support it", and just ignore it -a decision made on an 
option-by-option basis. 
   * if something unrecognised comes in as .must() the FS must raise an 
exception "I don't recognise that".
   * if something it does recognise as a .must() which it knows it doesn't 
support and which it MUST be able to, then it MUST raise an exception.
   
   Example: vectored IO will only work if the FS supports seek, which ftp 
doesn't. Assuming we add an option to say "I Want vectored IO" then all 
filesystems will get this out the box as a well known-and-supported feature, 
but ftp will need some explicit code to say "fail if vectored IO is requested"



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 538993)
Time Spent: 8h 50m  (was: 8h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=538736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-538736
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 20/Jan/21 22:30
Start Date: 20/Jan/21 22:30
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-763994374


   correct.   
   must() == raise an error if not recognised. 
   opt() entirely optional
   
   note that createFile() has the same api/rules/ common builder codebase. i 
hope to add some options there for the stores to say "skip all probes for paths 
existing, being a dir, having a parent..." goal is max performance writes on 
paths where the writer knows it is building up a dir tree from scratch, or 
using unique names. in particular: flink/spark checkpoints



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 538736)
Time Spent: 8h 40m  (was: 8.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536834
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 16/Jan/21 17:17
Start Date: 16/Jan/21 17:17
Worklog Time Spent: 10m 
  Work Description: ThomasMarquardt commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-761600086


   OK, so must() has already shipped, in that case sure, it should stay.  I 
still have a question.  If I pass an option via must(), can it be ignored by 
the implementation?  I read your documentation, specifically, "This means that 
it is not a requirement for the stores to actually read the the read policy or 
file length values and use them when opening files."  To me, this says that 
options passed via must() can be ignored.  If this is true, then what is the 
difference between must() and opt()?  Is it that must() can result in an 
unsupported exception, but opt() never results in an unsupported exception?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 536834)
Time Spent: 8.5h  (was: 8h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536797
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 16/Jan/21 13:17
Start Date: 16/Jan/21 13:17
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-761561997


   1. we've already shipped that
   2. it's in createFile too
   
   imagine in future you want to do something more than a hint. Case in point: 
the SQL API in AWS S3. there's no way an app could issue SQL commands which 
include transforming the response without knowing whether or not they are 
processed. hence `must()`. 
   
   how about i clarify this in the docs and recommend use of opt() except in 
such special cases?
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 536797)
Time Spent: 8h 20m  (was: 8h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-15 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536623
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 15/Jan/21 20:10
Start Date: 15/Jan/21 20:10
Worklog Time Spent: 10m 
  Work Description: ThomasMarquardt commented on a change in pull request 
#2584:
URL: https://github.com/apache/hadoop/pull/2584#discussion_r558505973



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
##
@@ -4655,7 +4656,7 @@ public FutureDataInputStreamBuilder openFile(PathHandle 
pathHandle)
   final OpenFileParameters parameters) throws IOException {
 AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(

Review comment:
   This rejectUnknownMandatoryKeys function would not be necessary if the 
mandatory keys were strongly typed fields.

##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSBuilder.java
##
@@ -61,6 +61,13 @@
*/
   B opt(@Nonnull String key, float value);
 
+  /**
+   * Set optional long parameter for the Builder.
+   *
+   * @see #opt(String, String)
+   */
+  B opt(@Nonnull String key, long value);
+

Review comment:
   We could use properties (getter and setter) for the standard options. 
We'd have one for buffer size, one for file length, etc.

##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,522 @@
+
+
+# `FileSystem.openFile()`/`FileContext.openFile()`
+
+This is a method provided by both FileSystem and FileContext for
+advanced file opening options and, where implemented, 
+an asynchrounous/lazy opening of a file.
+
+Creates a builder to open a file, supporting options
+both standard and filesystem specific. The return
+value of the `build()` call is a `Future`,
+which must be waited on. The file opening may be
+asynchronous, and it may actually be postponed (including
+permission/existence checks) until reads are actually
+performed.
+
+This API call was added to `FileSystem` and `FileContext` in
+Hadoop 3.3.0; it was tuned in Hadoop 3.3.1 as follows.
+
+* Added `opt(key, long)` and `must(key, long)`. 
+* Declared that `withFileStatus(null)` is allowed.
+* Declared that `withFileStatus(status)` only checks
+  the filename of the path, not the full path.
+  This is needed to support passthrough/mounted filesystems.
+* Added standard option keys.
+
+###   `FutureDataInputStreamBuilder openFile(Path 
path)`
+
+Creates a [`FutureDataInputStreamBuilder`](fsdatainputstreambuilder.html)
+to construct a operation to open the file at `path` for reading.
+
+When `build()` is invoked on the returned `FutureDataInputStreamBuilder` 
instance,
+the builder parameters are verified and
+`FileSystem.openFileWithOptions(Path, OpenFileParameters)` or
+`AbstractFileSystem.openFileWithOptions(Path, OpenFileParameters)` invoked.
+
+These protected methods returns a `CompletableFuture`
+which, when its `get()` method is called, either returns an input
+stream of the contents of opened file, or raises an exception.
+
+The base implementation of the `FileSystem.openFileWithOptions(PathHandle, 
OpenFileParameters)`
+ultimately invokes `FileSystem.open(Path, int)`.
+
+Thus the chain `FileSystem.openFile(path).build().get()` has the same 
preconditions
+and postconditions as `FileSystem.open(Path p, int bufferSize)`
+
+However, there is one difference which implementations are free to
+take advantage of: 
+
+The returned stream MAY implement a lazy open where file non-existence or
+access permission failures may not surface until the first `read()` of the
+actual data.
+
+The `openFile()` operation MAY check the state of the filesystem during its
+invocation, but as the state of the filesystem may change betwen this call and
+the actual `build()` and `get()` operations, this file-specific
+preconditions (file exists, file is readable, etc) MUST NOT be checked here.
+
+FileSystem implementations which do not implement `open(Path, int)`
+MAY postpone raising an `UnsupportedOperationException` until either the
+`FutureDataInputStreamBuilder.build()` or the subsequent `get()` call,
+else they MAY fail fast in the `openFile()` call.
+
+### Implementors notes
+
+The base implementation of `openFileWithOptions()` actually executes
+the `open(path)` operation synchronously, yet still returns the result
+or any failures in the `CompletableFuture<>`, so as to ensure that users
+code expecting this.
+
+Any filesystem where the time to open a file may be significant SHOULD
+execute it asynchronously by submitting the operation in some executor/thread
+pool. This is particularly recommended for object stores and other filesystems
+likely to be accessed over long-haul connections.
+
+Arbitrary filesystem-sp

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-15 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536449&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536449
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 15/Jan/21 12:22
Start Date: 15/Jan/21 12:22
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-760911137


   @ThomasMarquardt could you take a look @ this ?
   
   * I've updated the docs as suggested
   * proposed making the policy broader than just seek policy, so allowing 
stores to turn on whatever other tuning options they have, especially for file 
types they've profiled
   The goal there is rather than set cluster wide options which work well for 
some datatypes but are suboptimal for others, the app provides more information 
down.
   
   weakness there is that with multiple libraries working with Parquet data 
(spark, parquet.jar, iceberg, impala) it's not enough to declare the format. 
You'd really need to declare your app and version



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 536449)
Time Spent: 8h  (was: 7h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-15 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536448
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 15/Jan/21 12:18
Start Date: 15/Jan/21 12:18
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757007173


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 33s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 15 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 49s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  25m 51s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  22m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m 33s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 39s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  31m 37s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 31s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  12m 13s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 56s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 23s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   3m  1s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 6 new + 822 unchanged - 1 fixed = 828 total (was 
823)  |
   | +1 :green_heart: |  mvnsite  |   7m 25s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
 |  The patch has 14 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 28s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | -1 :x: |  javadoc  |   1m  3s | 
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  hadoop-common in the patch failed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  12m  1s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 41s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 22s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  7s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 30s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 17s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :gre

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536264&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536264
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 15/Jan/21 00:21
Start Date: 15/Jan/21 00:21
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-760560973


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 34s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 16 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 56s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m  3s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 30s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 58s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 45s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  26m 36s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 17s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 13s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  10m 40s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 41s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 23s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 59s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/4/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 3 new + 822 unchanged - 1 fixed = 825 total (was 
823)  |
   | +1 :green_heart: |  mvnsite  |   7m 25s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/4/artifact/out/whitespace-eol.txt)
 |  The patch has 16 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 19s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 55s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 42s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  11m 59s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 42s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 21s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  9s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 32s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 19s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 11s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   6m 41s |  |  hadoop-streaming in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 47s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 59s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 273m 24s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docke

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536032&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536032
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 14/Jan/21 14:58
Start Date: 14/Jan/21 14:58
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r557456802



##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return
+value of the `build()` call is a `Future`,
+which must be waited on. The file opening may be
+asynchronous, and it may actually be postponed (including
+permission/existence checks) until reads are actually
+performed.
+
+This API call was added to `FileSystem` and `FileContext` in
+Hadoop 3.3.0; it was tuned in Hadoop 3.3.1 as follows.
+
+* Support `opt(key, long)` and `must(key, long)`. 
+* Declare that `withFileStatus(null)` is allowed.
+* Declare that `withFileStatus(status)` only checks
+  the filename of the path, not the full path.
+  This is needed to support passthrough/mounted filesystems.
+
+
+###   `FSDataInputStreamBuilder openFile(Path 
path)`

Review comment:
   oops. Will do. I won't rename the fsdatainputstreambuilder.md file 
though -it's already shipped and I don't want to break links across versions. 
Will change all text references





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 536032)
Time Spent: 7.5h  (was: 7h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=536016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536016
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 14/Jan/21 14:26
Start Date: 14/Jan/21 14:26
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-738373147


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  26m 20s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 11 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 46s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 54s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 55s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 13s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m  3s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 32s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 11s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m  0s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 18s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   9m 54s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 41s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m  6s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m  7s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m  7s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 56s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/18/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 3 new + 796 unchanged - 1 fixed = 799 total (was 
797)  |
   | +1 :green_heart: |  mvnsite  |   6m 38s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/18/artifact/out/whitespace-eol.txt)
 |  The patch has 8 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 27s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m  8s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  10m 51s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 35s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 21s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  3s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 28s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   0m 56s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  13m 38s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 48s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 59s |  |  The patch does not 
generate ASF License warnings.  |
   |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=535811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-535811
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 14/Jan/21 00:00
Start Date: 14/Jan/21 00:00
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-759834285


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  25m 54s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 15 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  23m 30s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m  3s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 19s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 58s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 26s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  26m 15s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 34s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  10m 58s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 26s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 38s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 20s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 20s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 56s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 3 new + 821 unchanged - 1 fixed = 824 total (was 
822)  |
   | +1 :green_heart: |  mvnsite  |   7m 23s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  1s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/3/artifact/out/whitespace-eol.txt)
 |  The patch has 14 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 13s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  12m  4s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  3s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 23s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  5s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 31s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 15s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   0m 59s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   6m 37s |  |  hadoop-streaming in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 31s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 58s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 308m  8s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docke

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=535603&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-535603
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 13/Jan/21 18:19
Start Date: 13/Jan/21 18:19
Worklog Time Spent: 10m 
  Work Description: steveloughran edited a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-759582014


   I'm thinking we should be more ambitious in read policy than just "fadvise", 
because we can then use it as a declaration for the input streams to tune all 
their params, eg. buffer sizing, whether to do async prefetch. 
   
   Then we could allow stores to support not-just seek policies, but declare 
what you were planning to read, e.g. "parquet-bytebuffer", to mean "I'm reading 
parquet files through the bytebuffer positioned read API"
   
   ```
   openFile("s3a://datasets/set1/input.parquet).
 opt("fs.openfile.policy, "parquet-vectored, parquet,random")
.build().get()
   ```
   
   
   example` opt(fs.openfile.read.policy, "parquet-vectored, parquet, random")`  
to mean "optimise for parquet for vectored IO, then generic vectored IO, then 
generic random IO". Store implementors would get to make their own decisions as 
to what to set based on profiling &c. We'd need the applications to set policy 
on `openFile()` -so would need to know what names to use. That we can discuss 
with them,  maybe by predefining some options which *may* be supported



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 535603)
Time Spent: 7h  (was: 6h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=535586&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-535586
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 13/Jan/21 17:56
Start Date: 13/Jan/21 17:56
Worklog Time Spent: 10m 
  Work Description: steveloughran edited a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-759582014


   I'm thinking we should be more ambitious in read policy than just "fadvise", 
because we can then use it as a declaration for the input streams to tune all 
their params, eg. buffer sizing, whether to do async prefetch. 
   
   Then we could allow stores to support not-just seek policies, but declare 
what you were planning to read, e.g. "parquet-bytebuffer", to mean "I'm reading 
parquet files through the bytebuffer positioned read API"
   
   ```
   openFile("s3a://datasets/set1/input.parquet).
 opt("fs.openfile.policy, "parquet-vectored, impala, parquet,random")
.build().get()
   ```
   
   
   example` opt(fs.openfile.read.policy, "parquet-vectored, parquet, random")`  
to mean "optimise for impala for vectored IO, then generic vectored IO, then 
generic random IO". Store implementors would get to make their own decisions as 
to what to set based on profiling &c. We'd need the applications to set policy 
on `openFile()` -so would need to know what names to use. That we can discuss 
with them,  maybe by predefining some options which *may* be supported



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 535586)
Time Spent: 6h 40m  (was: 6.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=535587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-535587
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 13/Jan/21 17:56
Start Date: 13/Jan/21 17:56
Worklog Time Spent: 10m 
  Work Description: steveloughran edited a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-759582014


   I'm thinking we should be more ambitious in read policy than just "fadvise", 
because we can then use it as a declaration for the input streams to tune all 
their params, eg. buffer sizing, whether to do async prefetch. 
   
   Then we could allow stores to support not-just seek policies, but declare 
what you were planning to read, e.g. "parquet-bytebuffer", to mean "I'm reading 
parquet files through the bytebuffer positioned read API"
   
   ```
   openFile("s3a://datasets/set1/input.parquet).
 opt("fs.openfile.policy, "parquet-vectored, parquet,random")
.build().get()
   ```
   
   
   example` opt(fs.openfile.read.policy, "parquet-vectored, parquet, random")`  
to mean "optimise for impala for vectored IO, then generic vectored IO, then 
generic random IO". Store implementors would get to make their own decisions as 
to what to set based on profiling &c. We'd need the applications to set policy 
on `openFile()` -so would need to know what names to use. That we can discuss 
with them,  maybe by predefining some options which *may* be supported



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 535587)
Time Spent: 6h 50m  (was: 6h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=535551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-535551
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 13/Jan/21 16:56
Start Date: 13/Jan/21 16:56
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-759582014


   I'm thinking we should be more ambitious in read policy than just "fadvise", 
because we can then use it as a declaration for the input streams to tune all 
their params, eg. buffer sizing, whether to do async prefetch. 
   
   Then we could allow stores to support not-just seek policies, but declare 
what you were planning to read, e.g. "impala-bytebuffer", to mean "I'm using 
impala with the bytebuffer"
   
   example` opt(fs.openfile.read.policy, "impala-vectored, impala, 
vectored,random")`  to mean "optimise for impala for vectored IO, then generic 
vectored IO, then generic random IO". Store implementors would get to make 
their own decisions as to what to set based on profiling &c. We'd need the 
applications to set policy on `openFile()` -so would need to know what names to 
use. That we can discuss with them,  maybe by predefining some options which 
*may* be supported



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 535551)
Time Spent: 6.5h  (was: 6h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=533839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533839
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 10/Jan/21 04:24
Start Date: 10/Jan/21 04:24
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757007173


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 33s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 15 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 49s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  25m 51s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  22m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m 33s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 39s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  31m 37s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 31s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  12m 13s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 56s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 23s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   3m  1s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 6 new + 822 unchanged - 1 fixed = 828 total (was 
823)  |
   | +1 :green_heart: |  mvnsite  |   7m 25s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
 |  The patch has 14 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 28s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | -1 :x: |  javadoc  |   1m  3s | 
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  hadoop-common in the patch failed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  12m  1s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 41s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 22s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  7s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 30s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 17s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=533793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533793
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 10/Jan/21 04:19
Start Date: 10/Jan/21 04:19
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757344464


   style
   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:210:
FSBuilder propagateOptions(: 'FSBuilder' has incorrect indentation 
level 4, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:182:import
 static 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: 
Unused import - 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. 
[UnusedImports]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:220:
fileStatus = createStatus(path, fileLength, blockSize);: 'if' child has 
incorrect indentation level 8, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:221:
  }: 'if rcurly' has incorrect indentation level 6, expected level should 
be 4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:323:
private OpenFileInformation(:13: More than 7 parameters (found 10). 
[ParameterNumber]
   
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestS3AOpenFileOperation.java:119:
assertOpenFile(: 'assertOpenFile' has incorrect indentation level 4, 
expected level should be 6. [Indentation]
   ```
   
   javadocs
   ```
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:126:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:127:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:128:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:154:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:155:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:156:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:197:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:198:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:199:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:227:
 error: bad use of '>'
   [ERROR]

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=533464&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533464
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Jan/21 18:05
Start Date: 09/Jan/21 18:05
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757344464


   style
   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:210:
FSBuilder propagateOptions(: 'FSBuilder' has incorrect indentation 
level 4, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:182:import
 static 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: 
Unused import - 
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. 
[UnusedImports]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:220:
fileStatus = createStatus(path, fileLength, blockSize);: 'if' child has 
incorrect indentation level 8, expected level should be 6. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:221:
  }: 'if rcurly' has incorrect indentation level 6, expected level should 
be 4. [Indentation]
   
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:323:
private OpenFileInformation(:13: More than 7 parameters (found 10). 
[ParameterNumber]
   
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestS3AOpenFileOperation.java:119:
assertOpenFile(: 'assertOpenFile' has incorrect indentation level 4, 
expected level should be 6. [Indentation]
   ```
   
   javadocs
   ```
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:126:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:127:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:128:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:154:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:155:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java:156:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:197:
 error: bad use of '>'
   [ERROR]*   fs.example.s3a.option => s3a:option
   [ERROR]   ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:198:
 error: bad use of '>'
   [ERROR]*   fs.example.fs.io.policy => s3a.io.policy
   [ERROR] ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:199:
 error: bad use of '>'
   [ERROR]*   fs.example.something => something
   [ERROR]  ^
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2584/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FutureIO.java:227:
 error: bad use of '>'
   [ERROR]

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=533238&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533238
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 08/Jan/21 21:30
Start Date: 08/Jan/21 21:30
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757007173


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 33s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 15 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 49s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  25m 51s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  22m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m 33s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   7m 39s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  31m 37s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 52s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m 32s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 31s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  12m 13s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 56s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 23s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   3m  1s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 6 new + 822 unchanged - 1 fixed = 828 total (was 
823)  |
   | +1 :green_heart: |  mvnsite  |   7m 25s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
 |  The patch has 14 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 28s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | -1 :x: |  javadoc  |   1m  3s | 
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  hadoop-common in the patch failed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.  |
   | +1 :green_heart: |  javadoc  |   6m 36s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  12m  1s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 41s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 22s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  7s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 30s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  12m 17s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=531165&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531165
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 05/Jan/21 12:45
Start Date: 05/Jan/21 12:45
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-754613375


   MR client not compiling; not seeing useful information from yetus.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 531165)
Time Spent: 5h 40m  (was: 5.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=530757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530757
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 04/Jan/21 16:41
Start Date: 04/Jan/21 16:41
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-754082437


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 10 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   5m 15s |  |  Maven dependency ordering for branch  |
   | -1 :x: |  mvninstall  |   0m 23s | 
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-mvninstall-root.txt)
 |  root in trunk failed.  |
   | -1 :x: |  compile  |  18m 15s | 
[/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  root in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.  |
   | -1 :x: |  compile  |   0m 28s | 
[/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
 |  root in trunk failed with JDK Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.  |
   | +1 :green_heart: |  checkstyle  |   7m  9s |  |  trunk passed  |
   | -1 :x: |  mvnsite  |   0m 36s | 
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in trunk failed.  |
   | -1 :x: |  mvnsite  |   0m 21s | 
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
 |  hadoop-yarn-common in trunk failed.  |
   | +1 :green_heart: |  shadedclient  |  26m 40s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 36s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   4m 16s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   0m 44s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | -1 :x: |  findbugs  |   0m 20s | 
[/branch-findbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in trunk failed.  |
   | -1 :x: |  findbugs  |   0m 19s | 
[/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
 |  hadoop-yarn-common in trunk failed.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 51s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 12s | 
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 12s | 
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
 |  hadoop-yarn-common in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 22s | 
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapr

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=530679&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530679
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 04/Jan/21 14:09
Start Date: 04/Jan/21 14:09
Worklog Time Spent: 10m 
  Work Description: steveloughran opened a new pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584


   #2168 rebased to trunk
   
   1. Does not (yet) address Thomas's comments
   2. not retested



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 530679)
Time Spent: 5h 20m  (was: 5h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=530678&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530678
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 04/Jan/21 14:08
Start Date: 04/Jan/21 14:08
Worklog Time Spent: 10m 
  Work Description: steveloughran closed pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 530678)
Time Spent: 5h 10m  (was: 5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2021-01-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=530677&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530677
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 04/Jan/21 14:03
Start Date: 04/Jan/21 14:03
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-753992078


   going to close this PR and open a new one as I've rebased to trunk; will 
merge everything back into a single patch



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 530677)
Time Spent: 5h  (was: 4h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=522230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-522230
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 12:47
Start Date: 09/Dec/20 12:47
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-741748543


   @ThomasMarquardt -thanks for the feedback; will respond/update the PR
   
   I've been doing it as one PR as it lets me 
   * do interface and impl together, plus contract tests
   * rebase on a regular basis on a patch that's had little attention from 
reviewers
   * test it downstream in a PoC fork of parquet just to see what works/doesn't.
   
   FWIW passing in the file status/length makes a big difference in code which 
goes from list dir -> opening every file in it. Other than that, changing the 
seek policy is the big one. 
   
   Feel free to not review those s3a bits, except to look and see if there are 
things you'd want/want to avoid. open() has got a lot more complicated than it 
used to be. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 522230)
Time Spent: 4h 50m  (was: 4h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=58&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-58
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 12:41
Start Date: 09/Dec/20 12:41
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r539273879



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/AbstractFSBuilderImpl.java
##
@@ -46,7 +46,7 @@
  * 
  *   .opt("foofs:option.a", true)
  *   .opt("foofs:option.b", "value")
- *   .opt("barfs:cache", true)
+ *   .opt("fs.s3a.open.option.etag", "9fe4c37c25b")

Review comment:
   That's just an example of a non-standard one. No actual s3a support for 
this as it complicates life too much





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 58)
Time Spent: 4h 40m  (was: 4.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=57&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-57
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 12:40
Start Date: 09/Dec/20 12:40
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r539272915



##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
##
@@ -518,4 +522,94 @@ public String toString() {
 MD5MD5CRC,  // MD5 of block checksums, which are MD5 over chunk CRCs
 COMPOSITE_CRC  // Block/chunk-independent composite CRC
   }
+
+  /**
+   * The standard {@code openFile()} options.
+   */
+  @InterfaceAudience.Public
+  @InterfaceStability.Evolving
+  public static final class OpenFileOptions {

Review comment:
   did have that initially, then pulled back in. No real preference





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 57)
Time Spent: 4.5h  (was: 4h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=56&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-56
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 12:39
Start Date: 09/Dec/20 12:39
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r539272392



##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return

Review comment:
   The API lets you declare options that MUST be understood by the FS, vs 
those which can be ignored. All the new standard options fit into the "must" 
category as recognised options -they just don't have any sig
   
   ```java
   Future f = openFile(path)
 .must("fs.opt.seek-policy-option-name", "random, adaptive") 
 .opt("fs.azure.readahead", 312L)   // ignored on other stores. If you made 
it a must() opt then reading other stores MUST fail
 .build();
   
   FSDataInputStream is = f.get();
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 56)
Time Spent: 4h 20m  (was: 4h 10m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=522024&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-522024
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 03:30
Start Date: 09/Dec/20 03:30
Worklog Time Spent: 10m 
  Work Description: ThomasMarquardt commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r538788336



##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return
+value of the `build()` call is a `Future`,
+which must be waited on. The file opening may be
+asynchronous, and it may actually be postponed (including
+permission/existence checks) until reads are actually
+performed.
+
+This API call was added to `FileSystem` and `FileContext` in
+Hadoop 3.3.0; it was tuned in Hadoop 3.3.1 as follows.
+
+* Support `opt(key, long)` and `must(key, long)`. 
+* Declare that `withFileStatus(null)` is allowed.
+* Declare that `withFileStatus(status)` only checks
+  the filename of the path, not the full path.
+  This is needed to support passthrough/mounted filesystems.
+
+
+###   `FSDataInputStreamBuilder openFile(Path 
path)`
+
+Creates a [`FSDataInputStreamBuilder`](fsdatainputstreambuilder.html)
+to construct a operation to open the file at `path` for reading.
+
+When `build()` is invoked on the returned `FSDataInputStreamBuilder` instance,
+the builder parameters are verified and
+`openFileWithOptions(Path, OpenFileParameters)` invoked.

Review comment:
   Might be helpful to point out that openFileWithOptions is a method on 
FileSystem.

##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return
+value of the `build()` call is a `Future`,
+which must be waited on. The file opening may be
+asynchronous, and it may actually be postponed (including
+permission/existence checks) until reads are actually
+performed.
+
+This API call was added to `FileSystem` and `FileContext` in
+Hadoop 3.3.0; it was tuned in Hadoop 3.3.1 as follows.
+
+* Support `opt(key, long)` and `must(key, long)`. 
+* Declare that `withFileStatus(null)` is allowed.
+* Declare that `withFileStatus(status)` only checks
+  the filename of the path, not the full path.
+  This is needed to support passthrough/mounted filesystems.
+
+
+###   `FSDataInputStreamBuilder openFile(Path 
path)`
+
+Creates a [`FSDataInputStreamBuilder`](fsdatainputstreambuilder.html)
+to construct a operation to open the file at `path` for reading.
+
+When `build()` is invoked on the returned `FSDataInputStreamBuilder` instance,
+the builder parameters are verified and
+`openFileWithOptions(Path, OpenFileParameters)` invoked.
+
+This (protected) operation returns a `CompletableFuture`
+which, when its `get()` method is called, either returns an input
+stream of the contents of opened file, or raises an exception.
+
+The base implementation of the `openFileWithOptions(PathHandle, 
OpenFileParameters)`
+ultimately invokes `open(Path, int)`.
+
+Thus the chain `openFile(path).build().get()` has the same preconditions
+and postconditions as `open(Path p, int bufferSize)`
+
+However, there is one difference which implementations are free to
+take advantage of: 
+
+The returned stream MAY implement a lazy open where file non-existence or
+access permission failures may not surface until the first `read()` of the
+actual data.
+
+The `openFile()` operation may check the state of the filesystem during its
+invocation, but as the state of the filesystem may change betwen this call and
+the actual `build()` and `get()` operations, this file-specific
+preconditions (file exists, file is readable, etc) MUST NOT be checked here.
+
+FileSystem implementations which do not implement `open(Path, int)`
+MAY postpone raising an `UnsupportedOperationException` until either the
+`FSDataInputStreamBuilder.build()` or the subsequent `get()` call,
+else they MAY fail fast in the `openFile()` call.
+
+### Implementors notes
+
+The base implementation of `openFileWithOptions()` actually executes
+the `open(path)` operation synchronously, yet still returns the result

Review comment:
   I think it calls open(path, int).

##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return

Review comment:
   "supporting both standard and filesystem specific option

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=521972&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-521972
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 09/Dec/20 00:56
Start Date: 09/Dec/20 00:56
Worklog Time Spent: 10m 
  Work Description: ThomasMarquardt commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-741352394


   Hindsight is 20-20, but it might still be worthwhile to break this up into 
multiple PRs: 1) the hadoop-common changes with the default implementation, 2) 
the consumers (avro, distcp, mapreduce, yarn, etc), and 3) updates to the 
filesystem implementations (AWS, etc).  That should help the reviewers to focus.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 521972)
Time Spent: 4h  (was: 3h 50m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=521863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-521863
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 08/Dec/20 20:33
Start Date: 08/Dec/20 20:33
Worklog Time Spent: 10m 
  Work Description: ThomasMarquardt commented on a change in pull request 
#2168:
URL: https://github.com/apache/hadoop/pull/2168#discussion_r538786090



##
File path: 
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/openfile.md
##
@@ -0,0 +1,290 @@
+
+
+# `openFile()`
+
+Create a builder to open a file, supporting options
+both standard and filesystem specific. The return

Review comment:
   "supporting both standard and filesystem specific options." 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 521863)
Time Spent: 3h 50m  (was: 3h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=521118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-521118
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 07/Dec/20 12:12
Start Date: 07/Dec/20 12:12
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-739880426


   The latest patch rounds things off. This thing is ready to go in. 
   * We now have the option to specify the start and end of splits; the input 
formats in the MR client do this.
   * everywhere in the code where we explicitly download sequential datasets 
request sequential IO. (actually, I've just realised `hadoop fs -head ` 
should request random IO as well as declare split lengths...we don't want a 
full GET).
   
   its important that FS implementations don't rely on split length to set max 
file len, because splits are allowed to overrun to ensure a whole record/block 
is read. Apps which pass split info down to worker processes (hive &c) need to 
pass in file size too if they want to save the HEAD request. It could still be 
used by the input streams if they can think of a way 
   
   1. For sequential IO: end of content length = min(split-end, file-length) 
for that initial request,
   2 For random IO, assume it's the initial EOF. 
   
   because openFile() declares FNFEs can be delayed until reads, we could also 
see if we could do an async HEAD request while processing that first GET/HEAD, 
so have the final file length without blocking. That would make streams more 
complex —at least now we have the option.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 521118)
Time Spent: 3h 40m  (was: 3.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=519897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-519897
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 03/Dec/20 22:25
Start Date: 03/Dec/20 22:25
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-738373147


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  26m 20s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 11 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 46s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 54s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 55s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 13s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m  3s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   6m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 32s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m 11s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m  0s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 18s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   9m 54s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 41s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m  6s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m  7s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m  7s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 56s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/18/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 3 new + 796 unchanged - 1 fixed = 799 total (was 
797)  |
   | +1 :green_heart: |  mvnsite  |   6m 38s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/18/artifact/out/whitespace-eol.txt)
 |  The patch has 8 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 27s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   5m  8s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   6m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |  10m 51s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 35s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 21s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   7m  3s |  |  hadoop-mapreduce-client-core in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 28s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |   0m 56s |  |  hadoop-mapreduce-examples in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  13m 38s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 48s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 59s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 28

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-12-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=519035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-519035
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 02/Dec/20 14:24
Start Date: 02/Dec/20 14:24
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-737261192


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 33s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 10 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 47s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m  6s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 51s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m  3s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 57s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 59s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 36s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 54s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   4m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   7m 34s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 39s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 26s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m  9s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m  3s |  |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   2m 57s |  |  root: The patch generated 
0 new + 763 unchanged - 1 fixed = 763 total (was 764)  |
   | +1 :green_heart: |  mvnsite  |   5m  0s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/17/artifact/out/whitespace-eol.txt)
 |  The patch has 9 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 21s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 54s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | -1 :x: |  javadoc  |   0m 43s | 
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/17/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
 |  
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 
88 unchanged - 0 fixed = 90 total (was 88)  |
   | +1 :green_heart: |  findbugs  |   8m 13s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 37s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 17s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   8m 34s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  11m 53s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 36s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 58

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-24 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=516299&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516299
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 24/Nov/20 18:31
Start Date: 24/Nov/20 18:31
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-733158507


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m 43s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 10 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 30s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 56s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 17s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  19m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   3m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 29s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 35s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 16s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   4m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 12s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   7m 23s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 33s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 16s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 42s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  21m 42s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 29s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  18m 29s |  |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   3m  4s |  |  root: The patch generated 
0 new + 763 unchanged - 1 fixed = 763 total (was 764)  |
   | +1 :green_heart: |  mvnsite  |   4m 30s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/16/artifact/out/whitespace-eol.txt)
 |  The patch has 9 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  17m 31s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 27s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | -1 :x: |  javadoc  |   0m 38s | 
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/16/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
 |  
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 
88 unchanged - 0 fixed = 90 total (was 88)  |
   | +1 :green_heart: |  findbugs  |   8m 29s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m 35s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m  5s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   8m 30s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  13m 16s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 44s |  |  hadoop-aws in the patch passed. 
 |
   | +1 :green_heart: |  asflicense  |   0m 47

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-24 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=516277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516277
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 24/Nov/20 17:57
Start Date: 24/Nov/20 17:57
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-733141433


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 10 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 46s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 53s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 59s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 48s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 35s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 36s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 37s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   4m 18s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 18s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   7m 35s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 39s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 27s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 10s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 46s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 46s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   3m  3s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/15/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 10 new + 765 unchanged - 1 fixed = 775 total (was 
766)  |
   | +1 :green_heart: |  mvnsite  |   4m 47s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/15/artifact/out/whitespace-eol.txt)
 |  The patch has 9 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 26s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | -1 :x: |  javadoc  |   1m  5s | 
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/15/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 7 new + 92 unchanged 
- 7 fixed = 99 total (was 99)  |
   | -1 :x: |  javadoc  |   0m 42s | 
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/15/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
 |  
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 
88 unchanged - 0 fixed = 90 total (was 88)  |
   | +1 :green_heart: |  findbugs  |   8m 16s |  |  the

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-24 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=516147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516147
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 24/Nov/20 14:10
Start Date: 24/Nov/20 14:10
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-732997464


   latest patch follows on from HADOOP-17318 and passes in status when opening 
.pending and .pendingset files.
   
   for the magic committer this saves 1 HEAD per file in task commit; 1 HEAD 
per task in job commit
   
   For max speedup/scale we'd need to also adopt incremental listing. Later.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 516147)
Time Spent: 2h 50m  (was: 2h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-24 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=516110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516110
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 24/Nov/20 13:22
Start Date: 24/Nov/20 13:22
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-732969746


   looking @ jackson code, MismatchedInputException covers a lot of cases -and 
I don't want to do anything "clever" about choosing specific IOEs based on the 
exception text.
   
   Modifying test to look for a PathIOException rather than EOFException



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 516110)
Time Spent: 2h 40m  (was: 2.5h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-23 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=515567&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515567
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 23/Nov/20 14:40
Start Date: 23/Nov/20 14:40
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-732204082


   legit failure of `hadoop.util.TestJsonSerialization`. 
   
   As well as fixing, maybe for  the serializer, map MismatchedInputException 
to EOFException for better handling up the stack
   
   ```
   Error Message
   `file:/tmp/Keyval3692109174763841309.json': Failed to read JSON file 
com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map 
due to end-of-input
at [Source: 
(org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream); line: 1, 
column: 0]: No content to map due to end-of-input
at [Source: 
(org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream); line: 1, 
column: 0]
   Stacktrace
   org.apache.hadoop.fs.PathIOException: 
   `file:/tmp/Keyval3692109174763841309.json': Failed to read JSON file 
com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map 
due to end-of-input
at [Source: 
(org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream); line: 1, 
column: 0]: No content to map due to end-of-input
at [Source: 
(org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream); line: 1, 
column: 0]
at 
org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:275)
at 
org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:246)
at 
org.apache.hadoop.util.TestJsonSerialization.lambda$testFileSystemEmptyPath$2(TestJsonSerialization.java:180)
at 
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
at 
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at 
org.apache.hadoop.util.TestJsonSerialization.testFileSystemEmptyPath(TestJsonSerialization.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
   Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No 
content to map due to end-of-input
at [Source: 
(org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream); line: 1, 
column: 0]
at 
com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
at 
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4360)
at 
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4205)
at 
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3251)
at 
org.apache.hadoop.util.JsonSerialization.fromJsonStream(JsonSerialization.java:163)
at 
org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:273)
... 18 more
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 515567)
Time Spent: 2.5h  (was: 2h 20m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-11-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=513274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513274
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 18/Nov/20 01:32
Start Date: 18/Nov/20 01:32
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-729318049


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 10 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m 59s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 54s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m  6s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 21s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 58s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   5m  2s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 39s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 55s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   4m 43s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 17s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   7m 35s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 39s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 26s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 20s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  19m 20s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 17s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 17s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   3m 27s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/14/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 10 new + 762 unchanged - 1 fixed = 772 total (was 
763)  |
   | +1 :green_heart: |  mvnsite  |   5m  1s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/14/artifact/out/whitespace-eol.txt)
 |  The patch has 9 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  15m 18s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | -1 :x: |  javadoc  |   1m  2s | 
[/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/14/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
 |  
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 7 new + 92 unchanged 
- 7 fixed = 99 total (was 99)  |
   | -1 :x: |  javadoc  |   0m 44s | 
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/14/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
 |  
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 
88 unchanged - 0 fixed = 90 total (was 88)  |
   | +1 :green_heart: |  findbugs  |   8m 16s |  |  the

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-10-26 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=504701&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-504701
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 26/Oct/20 12:34
Start Date: 26/Oct/20 12:34
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-716517539


   javadoc
   ```
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2168/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:48:
 warning: empty  tag
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 504701)
Time Spent: 2h 10m  (was: 2h)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-10-21 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=503424&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-503424
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 21/Oct/20 22:19
Start Date: 21/Oct/20 22:19
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-713909641


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  28m 47s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 7 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  11m 41s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 16s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 53s |  |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  17m 20s |  |  trunk passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 49s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 10s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  22m 13s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 47s |  |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   4m  0s |  |  trunk passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 15s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   6m 24s |  |  trunk passed  |
   | -0 :warning: |  patch  |   1m 37s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 36s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 20s |  |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javac  |  19m 20s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  17m 17s |  |  the patch passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  javac  |  17m 17s |  |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 46s | 
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/12/artifact/out/diff-checkstyle-root.txt)
 |  root: The patch generated 5 new + 441 unchanged - 1 fixed = 446 total (was 
442)  |
   | +1 :green_heart: |  mvnsite  |   4m  9s |  |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s | 
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/12/artifact/out/whitespace-eol.txt)
 |  The patch has 9 line(s) that end in whitespace. Use git apply 
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  14m 59s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 45s |  |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   0m 45s | 
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/12/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt)
 |  
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 
with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 
88 unchanged - 0 fixed = 89 total (was 88)  |
   | +1 :green_heart: |  findbugs  |   6m 52s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 41s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   4m 22s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |  13m 36s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 44s |  |  hadoop-aws in the patch passed.

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=501516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501516
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 16/Oct/20 10:54
Start Date: 16/Oct/20 10:54
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-709975189


   I'm wondering if on distcp I should leave off the file length.
   
   why so: if someone updates a file between distcp being scheduled and it 
being executed, the length will be different. I know people aren't "meant" to 
do that, but inevitably people already do.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 501516)
Time Spent: 1h 50m  (was: 1h 40m)

> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows 
> the length of the file to be declared. If set, *no check for the existence of 
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than 
> only S3AFileStatus -and not check that the path matches the path being 
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to 
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-10-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=496415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496415
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 07/Oct/20 09:33
Start Date: 07/Oct/20 09:33
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus removed a comment on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-695110140


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 31s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 
7 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   3m 19s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m 15s |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 25s |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  17m 16s |  trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 48s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m  0s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 34s |  branch has no errors when 
building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 46s |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   4m  0s |  trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m  0s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   6m 22s |  trunk passed  |
   | -0 :warning: |  patch  |   2m 22s |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 26s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 33s |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 57s |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javac  |  18m 57s |  the patch passed  |
   | +1 :green_heart: |  compile  |  16m 52s |  the patch passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  javac  |  16m 52s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 51s |  root: The patch generated 7 new 
+ 441 unchanged - 1 fixed = 448 total (was 442)  |
   | +1 :green_heart: |  mvnsite  |   4m  6s |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s |  The patch has 9 line(s) that end in 
whitespace. Use git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  14m 16s |  patch has no errors when 
building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 44s |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   3m 59s |  the patch passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   6m 54s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  |   9m 22s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  unit  |   4m 17s |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |  12m  1s |  hadoop-distcp in the patch passed. 
 |
   | +1 :green_heart: |  unit  |   1m 43s |  hadoop-aws in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 209m 19s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractOpen |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/10/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2168 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux e54c99411ae6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |

[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally

2020-09-18 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=486409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486409
 ]

ASF GitHub Bot logged work on HADOOP-16202:
---

Author: ASF GitHub Bot
Created on: 18/Sep/20 22:15
Start Date: 18/Sep/20 22:15
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-695110140


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 31s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files 
found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 
7 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   3m 19s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m 15s |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 25s |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  17m 16s |  trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   2m 48s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m  0s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 34s |  branch has no errors when 
building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 46s |  trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   4m  0s |  trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m  0s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   6m 22s |  trunk passed  |
   | -0 :warning: |  patch  |   2m 22s |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 26s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 33s |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 57s |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javac  |  18m 57s |  the patch passed  |
   | +1 :green_heart: |  compile  |  16m 52s |  the patch passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  javac  |  16m 52s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   2m 51s |  root: The patch generated 7 new 
+ 441 unchanged - 1 fixed = 448 total (was 442)  |
   | +1 :green_heart: |  mvnsite  |   4m  6s |  the patch passed  |
   | -1 :x: |  whitespace  |   0m  0s |  The patch has 9 line(s) that end in 
whitespace. Use git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  shadedclient  |  14m 16s |  patch has no errors when 
building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 44s |  the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   3m 59s |  the patch passed with JDK 
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   6m 54s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  |   9m 22s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  unit  |   4m 17s |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |  12m  1s |  hadoop-distcp in the patch passed. 
 |
   | +1 :green_heart: |  unit  |   1m 43s |  hadoop-aws in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 209m 19s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractOpen |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2168/10/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2168 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux e54c99411ae6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Buil

89 matches

Mail list logo