[GitHub] [hadoop] hadoop-yetus commented on pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
hadoop-yetus commented on pull request #2368:
URL: https://github.com/apache/hadoop/pull/2368#issuecomment-707536847
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 33m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 14s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 14s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 105m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ec39d768a392 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3786d6c3cc |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/3/testReport/ |
| Max. process+thread count | 437 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HADOOP-17223) update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13
[ https://issues.apache.org/jira/browse/HADOOP-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17223: --- Issue Type: Bug (was: Improvement) > update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13 > - > > Key: HADOOP-17223 > URL: https://issues.apache.org/jira/browse/HADOOP-17223 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Pranav Bheda >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-17223.001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Update the dependencies > * org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.12 > * org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17223) update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13
[ https://issues.apache.org/jira/browse/HADOOP-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HADOOP-17223: -- Assignee: Pranav Bheda > update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13 > - > > Key: HADOOP-17223 > URL: https://issues.apache.org/jira/browse/HADOOP-17223 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Pranav Bheda >Assignee: Pranav Bheda >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-17223.001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Update the dependencies > * org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.12 > * org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn opened a new pull request #2382: HADOOP-17038 Support disabling buffered reads in ABFS positional reads.
anoopsjohn opened a new pull request #2382: URL: https://github.com/apache/hadoop/pull/2382 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[ https://issues.apache.org/jira/browse/HADOOP-17038?focusedWorklogId=499902&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499902 ] ASF GitHub Bot logged work on HADOOP-17038: --- Author: ASF GitHub Bot Created on: 13/Oct/20 08:04 Start Date: 13/Oct/20 08:04 Worklog Time Spent: 10m Work Description: anoopsjohn opened a new pull request #2382: URL: https://github.com/apache/hadoop/pull/2382 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 499902) Time Spent: 50m (was: 40m) > Support disabling buffered reads in ABFS positional reads > - > > Key: HADOOP-17038 > URL: https://issues.apache.org/jira/browse/HADOOP-17038 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Labels: HBase, abfsactive, pull-request-available > Attachments: HBase Perf Test Report.xlsx, screenshot-1.png > > Time Spent: 50m > Remaining Estimate: 0h > > Right now it will do a seek to the position , read and then seek back to the > old position. (As per the impl in the super class) > In HBase kind of workloads we rely mostly on short preads. (like 64 KB size > by default). So would be ideal to support a pure pos read API which will not > even keep the data in a buffer but will only read the required data as what > is asked for by the caller. (Not reading ahead more data as per the read size > config) > Allow an optional boolean config to be specified while opening file for read > using which buffered pread can be disabled. > FutureDataInputStreamBuilder openFile(Path path) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[ https://issues.apache.org/jira/browse/HADOOP-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212916#comment-17212916 ] Anoop Sam John commented on HADOOP-17038: - New PR based on the suggestion from [~ste...@apache.org]. Using new openFile API to disable buffered reads while preads. The API is marked InterfaceStability.Unstable as of now. Will this be changed ? Thanks for the suggestions. Tests passed in Azure ADL Gen2 premium storage account in East US. I have an HBase PE test results on a 3 node cluster. Will give that charts in a while. We see 2x gains. Will give cluster details and hbase file details. > Support disabling buffered reads in ABFS positional reads > - > > Key: HADOOP-17038 > URL: https://issues.apache.org/jira/browse/HADOOP-17038 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Labels: HBase, abfsactive, pull-request-available > Attachments: HBase Perf Test Report.xlsx, screenshot-1.png > > Time Spent: 50m > Remaining Estimate: 0h > > Right now it will do a seek to the position , read and then seek back to the > old position. (As per the impl in the super class) > In HBase kind of workloads we rely mostly on short preads. (like 64 KB size > by default). So would be ideal to support a pure pos read API which will not > even keep the data in a buffer but will only read the required data as what > is asked for by the caller. (Not reading ahead more data as per the read size > config) > Allow an optional boolean config to be specified while opening file for read > using which buffered pread can be disabled. > FutureDataInputStreamBuilder openFile(Path path) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2377: HDFS-15624. fix the function of setting quota by storage type
hadoop-yetus commented on pull request #2377:
URL: https://github.com/apache/hadoop/pull/2377#issuecomment-707585617
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 6m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 21s | | trunk passed |
| +1 :green_heart: | compile | 24m 43s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 19m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 51s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 17s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 45s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 7s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 56s | | the patch passed |
| +1 :green_heart: | compile | 24m 7s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 24m 7s | | the patch passed |
| +1 :green_heart: | compile | 19m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 19m 24s | | the patch passed |
| -0 :warning: | checkstyle | 3m 0s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2377/5/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 6 new + 734 unchanged - 1 fixed = 740 total (was
735) |
| +1 :green_heart: | mvnsite | 3m 30s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2377/5/artifact/out/whitespace-eol.txt)
| The patch has 1 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 16m 5s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 14s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 27s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 11m 25s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 153m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2377/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 11m 44s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2377/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | asflicense | 1m 37s |
[/patch-asflicense-problems.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2377/5/artifact/out/patch-asflicense-problems.txt)
| The patch generated 8 ASF License warnings. |
| | | 384m 29s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| | hadoop.hdfs.TestModTime |
| | hadoop.hdfs.server.namenode.ha.TestNNHealthCheck |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| |
[jira] [Work logged] (HADOOP-17223) update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13
[ https://issues.apache.org/jira/browse/HADOOP-17223?focusedWorklogId=499926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499926 ] ASF GitHub Bot logged work on HADOOP-17223: --- Author: ASF GitHub Bot Created on: 13/Oct/20 08:54 Start Date: 13/Oct/20 08:54 Worklog Time Spent: 10m Work Description: aajisaka merged pull request #2242: URL: https://github.com/apache/hadoop/pull/2242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 499926) Time Spent: 0.5h (was: 20m) > update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13 > - > > Key: HADOOP-17223 > URL: https://issues.apache.org/jira/browse/HADOOP-17223 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Pranav Bheda >Assignee: Pranav Bheda >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-17223.001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update the dependencies > * org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.12 > * org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2242: HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13
aajisaka merged pull request #2242: URL: https://github.com/apache/hadoop/pull/2242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17223) update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13
[ https://issues.apache.org/jira/browse/HADOOP-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17223: --- Summary: update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (was: update org.apache.httpcomponents:httpclient to 4.5.12 and httpcore to 4.4.13) > update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 > - > > Key: HADOOP-17223 > URL: https://issues.apache.org/jira/browse/HADOOP-17223 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Pranav Bheda >Assignee: Pranav Bheda >Priority: Blocker > Labels: pull-request-available > Attachments: HADOOP-17223.001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update the dependencies > * org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.12 > * org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17223) update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13
[ https://issues.apache.org/jira/browse/HADOOP-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17223: --- Fix Version/s: 3.4.0 3.3.1 3.2.2 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Merged the PR into trunk, branch-3.3, and branch-3.2. Thank you [~pranavbheda] for your contribution. > update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 > - > > Key: HADOOP-17223 > URL: https://issues.apache.org/jira/browse/HADOOP-17223 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Pranav Bheda >Assignee: Pranav Bheda >Priority: Blocker > Labels: pull-request-available > Fix For: 3.2.2, 3.3.1, 3.4.0 > > Attachments: HADOOP-17223.001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update the dependencies > * org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.12 > * org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
bshashikant commented on pull request #2370: URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707600727 @smengcl , would you like to retrigger the checks to ensure the snapshot tests work just fine. they seem to fail in last run bcoz of memory issue with the server. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2382: HADOOP-17038 Support disabling buffered reads in ABFS positional reads.
hadoop-yetus commented on pull request #2382:
URL: https://github.com/apache/hadoop/pull/2382#issuecomment-707610116
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 3s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 72m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2382 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e024edcdf59e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3786d6c3cc |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated messag
[jira] [Work logged] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[
https://issues.apache.org/jira/browse/HADOOP-17038?focusedWorklogId=499934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499934
]
ASF GitHub Bot logged work on HADOOP-17038:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 09:17
Start Date: 13/Oct/20 09:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2382:
URL: https://github.com/apache/hadoop/pull/2382#issuecomment-707610116
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 3s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 72m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2382 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e024edcdf59e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3786d6c3cc |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2382/1/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-t
[jira] [Commented] (HADOOP-16990) Update Mockserver
[ https://issues.apache.org/jira/browse/HADOOP-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212988#comment-17212988 ] Akira Ajisaka commented on HADOOP-16990: Thanks [~weichiu] for committing the patch. It seems branch-3.2 is not updated, so I cherry-picked the addendum patch to branch-3.2. > Update Mockserver > - > > Key: HADOOP-16990 > URL: https://issues.apache.org/jira/browse/HADOOP-16990 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Attila Doroszlai >Priority: Major > Fix For: 3.2.2, 3.3.1, 3.4.0, 3.1.5 > > Attachments: HADOOP-16990-branch-3.1.004.patch, > HADOOP-16990-branch-3.3.002.patch, HADOOP-16990.001.patch, > HDFS-15620-branch-3.3-addendum.patch > > > We are on Mockserver 3.9.2 which is more than 5 years old. Time to update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2380: HDFS-15626 TestWebHDFS.testLargeDirectory failing
steveloughran commented on pull request #2380: URL: https://github.com/apache/hadoop/pull/2380#issuecomment-707703361 +1. Merging to trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2380: HDFS-15626 TestWebHDFS.testLargeDirectory failing
steveloughran merged pull request #2380: URL: https://github.com/apache/hadoop/pull/2380 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #1857: HADOOP-16878. Copy command in FileUtil to throw an exception if the source and destination is the same
ayushtkn commented on pull request #1857: URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707729162 @steveloughran I have pushed the change, which I felt was required. Give a check, if you have some comments, I will address them as well. Test failures doesn't seems to be related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500048
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 13:14
Start Date: 13/Oct/20 13:14
Worklog Time Spent: 10m
Work Description: ayushtkn commented on pull request #1857:
URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707729162
@steveloughran I have pushed the change, which I felt was required. Give a
check, if you have some comments, I will address them as well.
Test failures doesn't seems to be related.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500048)
Time Spent: 50m (was: 40m)
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Attachments: hdfsTest.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2342: HADOOP-17288. Use shaded guava from thirdparty.
ayushtkn commented on a change in pull request #2342:
URL: https://github.com/apache/hadoop/pull/2342#discussion_r503965522
##
File path: Jenkinsfile
##
@@ -23,7 +23,7 @@ pipeline {
options {
buildDiscarder(logRotator(numToKeepStr: '5'))
-timeout (time: 20, unit: 'HOURS')
+timeout (time: 30, unit: 'HOURS')
Review comment:
Yes, this is intentional since the changes are in the parent pom, the
tests all of all the modules are executed, the build was getting timeout in 20
hours, so have temporarily increased. Will revert this before pushing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17288) Use shaded guava from thirdparty
[
https://issues.apache.org/jira/browse/HADOOP-17288?focusedWorklogId=500058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500058
]
ASF GitHub Bot logged work on HADOOP-17288:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 13:49
Start Date: 13/Oct/20 13:49
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2342:
URL: https://github.com/apache/hadoop/pull/2342#discussion_r503965522
##
File path: Jenkinsfile
##
@@ -23,7 +23,7 @@ pipeline {
options {
buildDiscarder(logRotator(numToKeepStr: '5'))
-timeout (time: 20, unit: 'HOURS')
+timeout (time: 30, unit: 'HOURS')
Review comment:
Yes, this is intentional since the changes are in the parent pom, the
tests all of all the modules are executed, the build was getting timeout in 20
hours, so have temporarily increased. Will revert this before pushing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500058)
Time Spent: 2h 10m (was: 2h)
> Use shaded guava from thirdparty
>
>
> Key: HADOOP-17288
> URL: https://issues.apache.org/jira/browse/HADOOP-17288
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Use the shaded version of guava in hadoop-thirdparty
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2342: HADOOP-17288. Use shaded guava from thirdparty.
ayushtkn commented on a change in pull request #2342: URL: https://github.com/apache/hadoop/pull/2342#discussion_r503966871 ## File path: hadoop-project/pom.xml ## @@ -2198,7 +2227,7 @@ - true + false Review comment: Yes, Since the thirdparty version being used is a snapshot version. If we don't disable it the build fails due to different timestamp with the snapshot version at jenkins(though it doesn't bother me at local). Once the thirdaparty version is released. Will get this back to `true` while updating the version This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17288) Use shaded guava from thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-17288?focusedWorklogId=500059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500059 ] ASF GitHub Bot logged work on HADOOP-17288: --- Author: ASF GitHub Bot Created on: 13/Oct/20 13:50 Start Date: 13/Oct/20 13:50 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #2342: URL: https://github.com/apache/hadoop/pull/2342#discussion_r503966871 ## File path: hadoop-project/pom.xml ## @@ -2198,7 +2227,7 @@ - true + false Review comment: Yes, Since the thirdparty version being used is a snapshot version. If we don't disable it the build fails due to different timestamp with the snapshot version at jenkins(though it doesn't bother me at local). Once the thirdaparty version is released. Will get this back to `true` while updating the version This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500059) Time Spent: 2h 20m (was: 2h 10m) > Use shaded guava from thirdparty > > > Key: HADOOP-17288 > URL: https://issues.apache.org/jira/browse/HADOOP-17288 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Use the shaded version of guava in hadoop-thirdparty -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2342: HADOOP-17288. Use shaded guava from thirdparty.
ayushtkn commented on a change in pull request #2342: URL: https://github.com/apache/hadoop/pull/2342#discussion_r503968035 ## File path: hadoop-common-project/hadoop-common/pom.xml ## @@ -49,6 +49,12 @@ hadoop-annotations compile + + org.apache.hadoop.thirdparty + hadoop-shaded-guava + compile Review comment: Thanx, Will update in next iteration. If no further comments. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17288) Use shaded guava from thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-17288?focusedWorklogId=500061&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500061 ] ASF GitHub Bot logged work on HADOOP-17288: --- Author: ASF GitHub Bot Created on: 13/Oct/20 13:53 Start Date: 13/Oct/20 13:53 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #2342: URL: https://github.com/apache/hadoop/pull/2342#discussion_r503968035 ## File path: hadoop-common-project/hadoop-common/pom.xml ## @@ -49,6 +49,12 @@ hadoop-annotations compile + + org.apache.hadoop.thirdparty + hadoop-shaded-guava + compile Review comment: Thanx, Will update in next iteration. If no further comments. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500061) Time Spent: 2.5h (was: 2h 20m) > Use shaded guava from thirdparty > > > Key: HADOOP-17288 > URL: https://issues.apache.org/jira/browse/HADOOP-17288 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Use the shaded version of guava in hadoop-thirdparty -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.
steveloughran commented on pull request #2323: URL: https://github.com/apache/hadoop/pull/2323#issuecomment-707756428 build failure was in native IO. full PR in #2324 didn't show this. I'd rebase this PR except it would complicate everything else too much This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16830) Add public IOStatistics API
[ https://issues.apache.org/jira/browse/HADOOP-16830?focusedWorklogId=500063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500063 ] ASF GitHub Bot logged work on HADOOP-16830: --- Author: ASF GitHub Bot Created on: 13/Oct/20 13:57 Start Date: 13/Oct/20 13:57 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2323: URL: https://github.com/apache/hadoop/pull/2323#issuecomment-707756428 build failure was in native IO. full PR in #2324 didn't show this. I'd rebase this PR except it would complicate everything else too much This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500063) Time Spent: 8h 10m (was: 8h) > Add public IOStatistics API > --- > > Key: HADOOP-16830 > URL: https://issues.apache.org/jira/browse/HADOOP-16830 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 8h 10m > Remaining Estimate: 0h > > Applications like to collect the statistics which specific operations take, > by collecting exactly those operations done during the execution of FS API > calls by their individual worker threads, and returning these to their job > driver > * S3A has a statistics API for some streams, but it's a non-standard one; > Impala &c can't use it > * FileSystem storage statistics are public, but as they aren't cross-thread, > they don't aggregate properly > Proposed > # A new IOStatistics interface to serve up statistics > # S3A to implement > # other stores to follow > # Pass-through from the usual wrapper classes (FS data input/output streams) > It's hard to think about how best to offer an API for operation context > stats, and how to actually implement. > ThreadLocal isn't enough because the helper threads need to update on the > thread local value of the instigator > My Initial PoC doesn't address that issue, but it shows what I'm thinking of -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2371: HADOOP-17258. Magic S3Guard Committer to overwrite existing pendingSet file on task commit
steveloughran closed pull request #2371: URL: https://github.com/apache/hadoop/pull/2371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17258) MagicS3GuardCommitter fails with `pendingset` already exists
[
https://issues.apache.org/jira/browse/HADOOP-17258?focusedWorklogId=500067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500067
]
ASF GitHub Bot logged work on HADOOP-17258:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 14:00
Start Date: 13/Oct/20 14:00
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #2371:
URL: https://github.com/apache/hadoop/pull/2371
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500067)
Time Spent: 4h 50m (was: 4h 40m)
> MagicS3GuardCommitter fails with `pendingset` already exists
>
>
> Key: HADOOP-17258
> URL: https://issues.apache.org/jira/browse/HADOOP-17258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> In `trunk/branch-3.3/branch-3.2`, `MagicS3GuardCommitter.innerCommitTask` has
> `false` at `pendingSet.save`.
> {code}
> try {
> pendingSet.save(getDestFS(), taskOutcomePath, false);
> } catch (IOException e) {
> LOG.warn("Failed to save task commit data to {} ",
> taskOutcomePath, e);
> abortPendingUploads(context, pendingSet.getCommits(), true);
> throw e;
> }
> {code}
> And, it can cause a job failure like the following.
> {code}
> WARN TaskSetManager: Lost task 1562.1 in stage 1.0 (TID 1788, 100.92.11.63,
> executor 26): org.apache.spark.SparkException: Task failed while writing rows.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:123)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/__magic/app-attempt-/task_20200911063607_0001_m_001562.pendingset
> already exists
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:761)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
> at
> org.apache.hadoop.util.JsonSerialization.save(JsonSerialization.java:269)
> at
> org.apache.hadoop.fs.s3a.commit.files.PendingSet.save(PendingSet.java:170)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.innerCommitTask(MagicS3GuardCommitter.java:220)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.commitTask(MagicS3GuardCommitter.java:165)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:244)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
> {code}
> {code}
> 20/09/11 07:44:38 ERROR TaskSetManager: Task
[GitHub] [hadoop] steveloughran closed pull request #1836: HADOOP-16646. Backport S3A enhancements and fixes from trunk to branch-3.2
steveloughran closed pull request #1836: URL: https://github.com/apache/hadoop/pull/1836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16646) Backport S3A enhancements and fixes from trunk to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16646?focusedWorklogId=500069&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500069 ] ASF GitHub Bot logged work on HADOOP-16646: --- Author: ASF GitHub Bot Created on: 13/Oct/20 14:02 Start Date: 13/Oct/20 14:02 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #1836: URL: https://github.com/apache/hadoop/pull/1836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500069) Remaining Estimate: 0h Time Spent: 10m > Backport S3A enhancements and fixes from trunk to branch-3.2 > > > Key: HADOOP-16646 > URL: https://issues.apache.org/jira/browse/HADOOP-16646 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3 >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Backport all the stable features from hadoop-aws in trunk to branch-3.2 > Note. we've already pulled most of these into CDP, so they have had > integration testing already done, though there may be some differences in > dependencies (guava, mockito etc) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16646) Backport S3A enhancements and fixes from trunk to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-16646: Labels: pull-request-available (was: ) > Backport S3A enhancements and fixes from trunk to branch-3.2 > > > Key: HADOOP-16646 > URL: https://issues.apache.org/jira/browse/HADOOP-16646 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3 >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Backport all the stable features from hadoop-aws in trunk to branch-3.2 > Note. we've already pulled most of these into CDP, so they have had > integration testing already done, though there may be some differences in > dependencies (guava, mockito etc) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
smengcl commented on pull request #2370: URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707766741 > @smengcl , would you like to retrigger the checks to ensure the snapshot tests work just fine. they seem to fail in last run bcoz of memory issue with the server. sure. will do This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
mukund-thakur commented on a change in pull request #2368:
URL: https://github.com/apache/hadoop/pull/2368#discussion_r504018690
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRandomRead.java
##
@@ -448,15 +477,119 @@ public void testRandomReadPerformance() throws Exception
{
ratio < maxAcceptableRatio);
}
+ /**
+ * With this test we should see a full buffer read being triggered in case
+ * alwaysReadBufferSize is on, else only the requested buffer size.
+ * Hence a seek done few bytes away from last read position will trigger
+ * a network read when alwaysReadBufferSize is off, whereas it will return
+ * from the internal buffer when it is on.
+ * Reading a full buffer size is the Gen1 behaviour.
+ * @throws Throwable
+ */
+ @Test
+ public void testAlwaysReadBufferSizeConfig() throws Throwable {
+testAlwaysReadBufferSizeConfig(false);
+testAlwaysReadBufferSizeConfig(true);
+ }
+
+ private void assertStatistics(AzureBlobFileSystem fs,
Review comment:
Why creating a new method here if we are just doing a passthrough?
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -447,4 +490,168 @@ public void testReadAheadManagerForSuccessfulReadAhead()
throws Exception {
checkEvictedStatus(inputStream, 0, true);
}
+ /**
+ * Test readahead with different config settings for request request size and
+ * readAhead block size
+ * @throws Exception
+ */
+ @Test
+ public void testDiffReadRequestSizeAndRAHBlockSize() throws Exception {
+// Set requestRequestSize = 4MB and readAheadBufferSize=8MB
+ReadBufferManager.getBufferManager()
+.testResetReadBufferManager(FOUR_MB,
INCREASED_READ_BUFFER_AGE_THRESHOLD);
+testReadAheadConfigs(FOUR_MB, TEST_READAHEAD_DEPTH_4, false, EIGHT_MB);
+
+// Test for requestRequestSize =16KB and readAheadBufferSize=16KB
+ReadBufferManager.getBufferManager()
+.testResetReadBufferManager(SIXTEEN_KB,
INCREASED_READ_BUFFER_AGE_THRESHOLD);
+AbfsInputStream inputStream = testReadAheadConfigs(SIXTEEN_KB,
+TEST_READAHEAD_DEPTH_2, true, SIXTEEN_KB);
+testReadAheads(inputStream, SIXTEEN_KB, SIXTEEN_KB);
+
+// Test for requestRequestSize =16KB and readAheadBufferSize=48KB
+ReadBufferManager.getBufferManager()
+.testResetReadBufferManager(FORTY_EIGHT_KB,
INCREASED_READ_BUFFER_AGE_THRESHOLD);
+inputStream = testReadAheadConfigs(SIXTEEN_KB, TEST_READAHEAD_DEPTH_2,
true,
+FORTY_EIGHT_KB);
+testReadAheads(inputStream, SIXTEEN_KB, FORTY_EIGHT_KB);
+
+// Test for requestRequestSize =48KB and readAheadBufferSize=16KB
+ReadBufferManager.getBufferManager()
+.testResetReadBufferManager(FORTY_EIGHT_KB,
INCREASED_READ_BUFFER_AGE_THRESHOLD);
+inputStream = testReadAheadConfigs(FORTY_EIGHT_KB, TEST_READAHEAD_DEPTH_2,
+true,
+SIXTEEN_KB);
+testReadAheads(inputStream, FORTY_EIGHT_KB, SIXTEEN_KB);
+ }
+
+
+ private void testReadAheads(AbfsInputStream inputStream,
+ int readRequestSize,
+ int readAheadRequestSize)
+ throws Exception {
+if (readRequestSize > readAheadRequestSize) {
+ readAheadRequestSize = readRequestSize;
+}
+
+byte[] firstReadBuffer = new byte[readRequestSize];
+byte[] secondReadBuffer = new byte[readAheadRequestSize];
+
+// get the expected bytes to compare
+byte[] expectedFirstReadAheadBufferContents = new byte[readRequestSize];
+byte[] expectedSecondReadAheadBufferContents = new
byte[readAheadRequestSize];
+getExpectedBufferData(0, readRequestSize,
expectedFirstReadAheadBufferContents);
+getExpectedBufferData(readRequestSize, readAheadRequestSize,
+expectedSecondReadAheadBufferContents);
+
+assertTrue("Read should be of exact requested size",
+ inputStream.read(firstReadBuffer, 0, readRequestSize) ==
readRequestSize);
+assertTrue("Data mismatch found in RAH1",
+Arrays.equals(firstReadBuffer,
+expectedFirstReadAheadBufferContents));
+
+
+assertTrue("Read should be of exact requested size",
+inputStream.read(secondReadBuffer, 0, readAheadRequestSize) ==
readAheadRequestSize);
+assertTrue("Data mismatch found in RAH2",
Review comment:
Better to use assert equals here inspite of assertTrue no?
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -447,4 +490,168 @@ public void testReadAheadManagerForSuccessfulReadAhead()
throws Exception {
checkEvictedStatus(inputStream, 0, true);
}
+ /**
+ * Test readahead with different config settings for request request size and
+ * readAhead block size
+ * @throws Exception
+ */
+ @Test
+ public void testDiffReadRequestSizeAndRAHBlockSize() throws Exception {
[GitHub] [hadoop] steveloughran commented on pull request #1857: HADOOP-16878. Copy command in FileUtil to throw an exception if the source and destination is the same
steveloughran commented on pull request #1857: URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707811423 yes, tests are unrelated. webhdfs one has been fixed; not sure about the checksum ones but I see them a lot... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500104
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:15
Start Date: 13/Oct/20 15:15
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #1857:
URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707811423
yes, tests are unrelated. webhdfs one has been fixed; not sure about the
checksum ones but I see them a lot...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500104)
Time Spent: 1h (was: 50m)
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Attachments: hdfsTest.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1857: HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
steveloughran merged pull request #1857: URL: https://github.com/apache/hadoop/pull/1857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500106
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:18
Start Date: 13/Oct/20 15:18
Worklog Time Spent: 10m
Work Description: steveloughran merged pull request #1857:
URL: https://github.com/apache/hadoop/pull/1857
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500106)
Time Spent: 1h 10m (was: 1h)
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16878:
Fix Version/s: 3.4.0
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16878:
Summary: FileUtil.copy() to throw IOException if the source and destination
are the same (was: Copy command in FileUtil to throw an exception if the
source and destination is the same)
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16878:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #1857: HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
steveloughran commented on pull request #1857: URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707817117 Merged to trunk. Not in branch-3.3 as there's enough of a diff in the TestDistributedFileSystem suite that the merge is failing. Thoughts? It'd be nice to backport, wouldn't it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500112
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:23
Start Date: 13/Oct/20 15:23
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #1857:
URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707817117
Merged to trunk. Not in branch-3.3 as there's enough of a diff in the
TestDistributedFileSystem suite that the merge is failing.
Thoughts? It'd be nice to backport, wouldn't it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500112)
Time Spent: 1h 20m (was: 1h 10m)
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[ https://issues.apache.org/jira/browse/HADOOP-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-17304: Attachment: HADOOP-17304.002.patch > KMS ACL: Allow DeleteKey Operation to Invalidate Cache > -- > > Key: HADOOP-17304 > URL: https://issues.apache.org/jira/browse/HADOOP-17304 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17304.001.patch, HADOOP-17304.002.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HADOOP-17208 send invalidate cache for key being deleted. The invalidate > cache operation itself requires ROLLOVER permission on the key. This ticket > is opened to fix the issue caught by TestKMS.testACLs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[ https://issues.apache.org/jira/browse/HADOOP-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213171#comment-17213171 ] Xiaoyu Yao commented on HADOOP-17304: - Thanks [~hexiaoqiao] for the review. Moved the const to KMSACLs as suggested in 002 patch. > KMS ACL: Allow DeleteKey Operation to Invalidate Cache > -- > > Key: HADOOP-17304 > URL: https://issues.apache.org/jira/browse/HADOOP-17304 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17304.001.patch, HADOOP-17304.002.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HADOOP-17208 send invalidate cache for key being deleted. The invalidate > cache operation itself requires ROLLOVER permission on the key. This ticket > is opened to fix the issue caught by TestKMS.testACLs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2369: HADOOP-17301. ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
hadoop-yetus removed a comment on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705474826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 71m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8b0a2d453311 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 735e85a1d20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500114
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:29
Start Date: 13/Oct/20 15:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705474826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 71m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8b0a2d453311 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 735e85a1d20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-S
[GitHub] [hadoop] steveloughran merged pull request #2369: HADOOP-17301. ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
steveloughran merged pull request #2369: URL: https://github.com/apache/hadoop/pull/2369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500115&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500115 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 15:30 Start Date: 13/Oct/20 15:30 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2369: URL: https://github.com/apache/hadoop/pull/2369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500115) Time Spent: 3h 10m (was: 3h) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500116 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 15:31 Start Date: 13/Oct/20 15:31 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707822816 +1, merged to trunk. @snvijaya -before I backport it, can you do a cherrypick of this into branch-3.3 and verify all the tests are good? thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500116) Time Spent: 3h 20m (was: 3h 10m) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2369: HADOOP-17301. ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
steveloughran commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707822816 +1, merged to trunk. @snvijaya -before I backport it, can you do a cherrypick of this into branch-3.3 and verify all the tests are good? thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17301) ABFS: read-ahead error reporting breaks buffer management
[ https://issues.apache.org/jira/browse/HADOOP-17301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17301: Summary: ABFS: read-ahead error reporting breaks buffer management (was: ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back) > ABFS: read-ahead error reporting breaks buffer management > - > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17301) ABFS: read-ahead error reporting breaks buffer management
[ https://issues.apache.org/jira/browse/HADOOP-17301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213177#comment-17213177 ] Steve Loughran commented on HADOOP-17301: - fixed in trunk, leaving open for backport to branch-3.3 > ABFS: read-ahead error reporting breaks buffer management > - > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn opened a new pull request #2383: HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
ayushtkn opened a new pull request #2383: URL: https://github.com/apache/hadoop/pull/2383 Backport to branch-3.3 https://issues.apache.org/jira/browse/HADOOP-16878 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500130
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:58
Start Date: 13/Oct/20 15:58
Worklog Time Spent: 10m
Work Description: ayushtkn opened a new pull request #2383:
URL: https://github.com/apache/hadoop/pull/2383
Backport to branch-3.3
https://issues.apache.org/jira/browse/HADOOP-16878
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500130)
Time Spent: 1.5h (was: 1h 20m)
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #1857: HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
ayushtkn commented on pull request #1857: URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707841768 Have raised https://github.com/apache/hadoop/pull/2383 for the backport This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500131&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500131
]
ASF GitHub Bot logged work on HADOOP-16878:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:59
Start Date: 13/Oct/20 15:59
Worklog Time Spent: 10m
Work Description: ayushtkn commented on pull request #1857:
URL: https://github.com/apache/hadoop/pull/1857#issuecomment-707841768
Have raised https://github.com/apache/hadoop/pull/2383 for the backport
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500131)
Time Spent: 1h 40m (was: 1.5h)
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
smengcl commented on pull request #2370: URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707848712 Resolved merge conflict in UT with HADOOP-16878 which is just merged to trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
smengcl edited a comment on pull request #2370: URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707766741 > @smengcl , would you like to retrigger the checks to ensure the snapshot tests work just fine. they seem to fail in last run bcoz of memory issue with the server. sure. will do triggered checks with an empty commit: https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-2370/3/ btw I ran `TestRenameWithSnapshots` and `TestSnapshotDeletion` locally and they passed. anyway, will wait for CI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
hadoop-yetus commented on pull request #2370:
URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707892562
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 43s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 16s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 14s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 0s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 14s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 99m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 185m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| |
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
|
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2370 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aea37738cba3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0507c4160f6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/3/testReport/ |
| Max. process+thread count | 4354 (vs. ulimit of 5500) |
|
[GitHub] [hadoop] smengcl commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
smengcl commented on pull request #2370: URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707910536 The last CI run looks good. Will merge in a minute. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl merged pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
smengcl merged pull request #2370: URL: https://github.com/apache/hadoop/pull/2370 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[
https://issues.apache.org/jira/browse/HADOOP-17304?focusedWorklogId=500214&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500214
]
ASF GitHub Bot logged work on HADOOP-17304:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 18:01
Start Date: 13/Oct/20 18:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2381:
URL: https://github.com/apache/hadoop/pull/2381#issuecomment-707913696
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 55s | | trunk passed |
| +1 :green_heart: | compile | 21m 11s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 48s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 56s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 48s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 45s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 20m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 20m 26s | | the patch passed |
| +1 :green_heart: | compile | 17m 39s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 17m 39s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 52s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 51s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 34s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 156m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2381/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2381 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e86230d0b9b7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 59f01a548e4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2381/2/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/j
[GitHub] [hadoop] hadoop-yetus commented on pull request #2381: HADOOP-17304. KMS ACL: Allow DeleteKey Operation to Invalidate Cache.
hadoop-yetus commented on pull request #2381:
URL: https://github.com/apache/hadoop/pull/2381#issuecomment-707913696
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 55s | | trunk passed |
| +1 :green_heart: | compile | 21m 11s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 48s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 56s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 48s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 45s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 20m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 20m 26s | | the patch passed |
| +1 :green_heart: | compile | 17m 39s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 17m 39s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 52s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 51s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 34s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 156m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2381/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2381 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e86230d0b9b7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 59f01a548e4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2381/2/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2381/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[jira] [Commented] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[
https://issues.apache.org/jira/browse/HADOOP-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213285#comment-17213285
]
Hadoop QA commented on HADOOP-17304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
17s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
50s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
16s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
45s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 22s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
51s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
6s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
6s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
23s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
23s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 54s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:gr
[GitHub] [hadoop] hadoop-yetus commented on pull request #2370: HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled
hadoop-yetus commented on pull request #2370:
URL: https://github.com/apache/hadoop/pull/2370#issuecomment-707949432
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 26s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 0s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 57s | | trunk passed |
| -0 :warning: | patch | 3m 17s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 1m 3s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 2s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 4s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 95m 0s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 178m 40s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2370 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 214291bf7207 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bd8cf7fd4ce |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2370/4/testReport/ |
| Max. process+thread count | 3984 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-
[GitHub] [hadoop] hadoop-yetus commented on pull request #2343: HADOOP-17289 ABFS: Fixing the test case ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
hadoop-yetus commented on pull request #2343:
URL: https://github.com/apache/hadoop/pull/2343#issuecomment-707958801
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 58s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 16s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
| -0 :warning: | patch | 1m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 8s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 1m 28s |
[/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2343/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 72m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.azure.TestBlobMetadata |
| | hadoop.fs.azure.TestNativeAzureFileSystemConcurrency |
| | hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck |
| | hadoop.fs.azure.TestWasbFsck |
| | hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked |
| | hadoop.fs.azure.TestOutOfBandAzureBlobOperations |
| | hadoop.fs.azure.TestNativeAzureFileSystemContractMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemMocked |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2343/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2343 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux be1a6e361e33 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a308a1ec22b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-
[jira] [Work logged] (HADOOP-17289) ABFS: Testcase failure ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17289?focusedWorklogId=500249&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500249
]
ASF GitHub Bot logged work on HADOOP-17289:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 19:26
Start Date: 13/Oct/20 19:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2343:
URL: https://github.com/apache/hadoop/pull/2343#issuecomment-707958801
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 58s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 16s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
| -0 :warning: | patch | 1m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 8s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 1m 28s |
[/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2343/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 72m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.azure.TestBlobMetadata |
| | hadoop.fs.azure.TestNativeAzureFileSystemConcurrency |
| | hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck |
| | hadoop.fs.azure.TestWasbFsck |
| | hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked |
| | hadoop.fs.azure.TestOutOfBandAzureBlobOperations |
| | hadoop.fs.azure.TestNativeAzureFileSystemContractMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemMocked |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2343/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2343 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux be1a6e361e33 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
[GitHub] [hadoop] hadoop-yetus commented on pull request #2383: HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
hadoop-yetus commented on pull request #2383: URL: https://github.com/apache/hadoop/pull/2383#issuecomment-707997222 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 29m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +0 :ok: | mvndep | 3m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 23s | branch-3.3 passed | | +1 :green_heart: | compile | 15m 26s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 2m 41s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 2m 53s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 20m 59s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 0s | branch-3.3 passed | | +0 :ok: | spotbugs | 3m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 11s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 58s | the patch passed | | +1 :green_heart: | compile | 14m 52s | the patch passed | | +1 :green_heart: | javac | 14m 52s | the patch passed | | +1 :green_heart: | checkstyle | 2m 40s | the patch passed | | +1 :green_heart: | mvnsite | 2m 51s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 1s | the patch passed | | +1 :green_heart: | findbugs | 5m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 0s | hadoop-common in the patch passed. | | -1 :x: | unit | 118m 59s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | | | | 282m 50s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.TestGroupsCaching | | | hadoop.hdfs.web.TestWebHDFS | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2383 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 14d21fe55d04 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 351b149 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/testReport/ | | Max. process+thread count | 3680 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?focusedWorklogId=500282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500282 ] ASF GitHub Bot logged work on HADOOP-16878: --- Author: ASF GitHub Bot Created on: 13/Oct/20 20:42 Start Date: 13/Oct/20 20:42 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2383: URL: https://github.com/apache/hadoop/pull/2383#issuecomment-707997222 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 29m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +0 :ok: | mvndep | 3m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 23s | branch-3.3 passed | | +1 :green_heart: | compile | 15m 26s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 2m 41s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 2m 53s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 20m 59s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 0s | branch-3.3 passed | | +0 :ok: | spotbugs | 3m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 11s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 58s | the patch passed | | +1 :green_heart: | compile | 14m 52s | the patch passed | | +1 :green_heart: | javac | 14m 52s | the patch passed | | +1 :green_heart: | checkstyle | 2m 40s | the patch passed | | +1 :green_heart: | mvnsite | 2m 51s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 1s | the patch passed | | +1 :green_heart: | findbugs | 5m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 0s | hadoop-common in the patch passed. | | -1 :x: | unit | 118m 59s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | | | | 282m 50s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.TestGroupsCaching | | | hadoop.hdfs.web.TestWebHDFS | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2383 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 14d21fe55d04 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 351b149 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/testReport/ | | Max. process+thread count | 3680 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2383/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time
[GitHub] [hadoop] swaroopak closed pull request #2372: MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
swaroopak closed pull request #2372: URL: https://github.com/apache/hadoop/pull/2372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[ https://issues.apache.org/jira/browse/HADOOP-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213580#comment-17213580 ] Xiaoqiao He commented on HADOOP-17304: -- LGTM, +1 for [^HADOOP-17304.002.patch]. Will commit if no objection. [~ayushtkn] would you like to have another review? > KMS ACL: Allow DeleteKey Operation to Invalidate Cache > -- > > Key: HADOOP-17304 > URL: https://issues.apache.org/jira/browse/HADOOP-17304 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17304.001.patch, HADOOP-17304.002.patch > > Time Spent: 20m > Remaining Estimate: 0h > > HADOOP-17208 send invalidate cache for key being deleted. The invalidate > cache operation itself requires ROLLOVER permission on the key. This ticket > is opened to fix the issue caught by TestKMS.testACLs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17304) KMS ACL: Allow DeleteKey Operation to Invalidate Cache
[ https://issues.apache.org/jira/browse/HADOOP-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213619#comment-17213619 ] Ayush Saxena commented on HADOOP-17304: --- Well the patch itself looks good. [~hexiaoqiao] give a check to the compatibility guidelines, it shouldn't break any downstream code, Changing behaviour of/expectation from a public API is an incompatible change. If everything is cool in that side. Things should be good to go. > KMS ACL: Allow DeleteKey Operation to Invalidate Cache > -- > > Key: HADOOP-17304 > URL: https://issues.apache.org/jira/browse/HADOOP-17304 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17304.001.patch, HADOOP-17304.002.patch > > Time Spent: 20m > Remaining Estimate: 0h > > HADOOP-17208 send invalidate cache for key being deleted. The invalidate > cache operation itself requires ROLLOVER permission on the key. This ticket > is opened to fix the issue caught by TestKMS.testACLs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #2337: ABFS: Backport ABFS updates from trunk to Branch 3.3
snvijaya commented on pull request #2337: URL: https://github.com/apache/hadoop/pull/2337#issuecomment-708172045 Test re-runs. This time including configs needed to trigger SAS and checkaccess tests. ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428520 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java ## @@ -22,8 +22,9 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4SafeDecompressor; Review comment: ``` import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4SafeDecompressor; import org.apache.hadoop.io.compress.Decompressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428324 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java ## @@ -22,9 +22,10 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4Compressor; Review comment: They are already in same block, no? ```java import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4Compressor; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.compress.Compressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[ https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500451&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500451 ] ASF GitHub Bot logged work on HADOOP-17292: --- Author: ASF GitHub Bot Created on: 14/Oct/20 06:22 Start Date: 14/Oct/20 06:22 Worklog Time Spent: 10m Work Description: viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428520 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java ## @@ -22,8 +22,9 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4SafeDecompressor; Review comment: ``` import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4SafeDecompressor; import org.apache.hadoop.io.compress.Decompressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500451) Time Spent: 3h 10m (was: 3h) > Using lz4-java in Lz4Codec > -- > > Key: HADOOP-17292 > URL: https://issues.apache.org/jira/browse/HADOOP-17292 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > In Hadoop, we use native libs for lz4 codec which has several disadvantages: > It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and > they have to be installed separately on each node of the clusters, container > images, or local test environments which adds huge complexities from > deployment point of view. In some environments, it requires compiling the > natives from sources which is non-trivial. Also, this approach is platform > dependent; the binary may not work in different platform, so it requires > recompilation. > It requires extra configuration of java.library.path to load the natives, and > it results higher application deployment and maintenance cost for users. > Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which > is JNI-based implementation. It contains native binaries in jar file, and it > can automatically load the native binaries into JVM from jar without any > setup. If a native implementation can not be found for a platform, it can > fallback to pure-java implementation of lz4. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[ https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500450 ] ASF GitHub Bot logged work on HADOOP-17292: --- Author: ASF GitHub Bot Created on: 14/Oct/20 06:22 Start Date: 14/Oct/20 06:22 Worklog Time Spent: 10m Work Description: viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428324 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java ## @@ -22,9 +22,10 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4Compressor; Review comment: They are already in same block, no? ```java import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4Compressor; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.compress.Compressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500450) Time Spent: 3h (was: 2h 50m) > Using lz4-java in Lz4Codec > -- > > Key: HADOOP-17292 > URL: https://issues.apache.org/jira/browse/HADOOP-17292 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > In Hadoop, we use native libs for lz4 codec which has several disadvantages: > It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and > they have to be installed separately on each node of the clusters, container > images, or local test environments which adds huge complexities from > deployment point of view. In some environments, it requires compiling the > natives from sources which is non-trivial. Also, this approach is platform > dependent; the binary may not work in different platform, so it requires > recompilation. > It requires extra configuration of java.library.path to load the natives, and > it results higher application deployment and maintenance cost for users. > Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which > is JNI-based implementation. It contains native binaries in jar file, and it > can automatically load the native binaries into JVM from jar without any > setup. If a native implementation can not be found for a platform, it can > fallback to pure-java implementation of lz4. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428520 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java ## @@ -22,8 +22,9 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4SafeDecompressor; Review comment: Already in same block? ```java import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4SafeDecompressor; import org.apache.hadoop.io.compress.Decompressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[ https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500453&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500453 ] ASF GitHub Bot logged work on HADOOP-17292: --- Author: ASF GitHub Bot Created on: 14/Oct/20 06:23 Start Date: 14/Oct/20 06:23 Worklog Time Spent: 10m Work Description: viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504428520 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java ## @@ -22,8 +22,9 @@ import java.nio.Buffer; import java.nio.ByteBuffer; +import net.jpountz.lz4.LZ4Factory; +import net.jpountz.lz4.LZ4SafeDecompressor; Review comment: Already in same block? ```java import net.jpountz.lz4.LZ4Factory; import net.jpountz.lz4.LZ4SafeDecompressor; import org.apache.hadoop.io.compress.Decompressor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500453) Time Spent: 3h 20m (was: 3h 10m) > Using lz4-java in Lz4Codec > -- > > Key: HADOOP-17292 > URL: https://issues.apache.org/jira/browse/HADOOP-17292 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > In Hadoop, we use native libs for lz4 codec which has several disadvantages: > It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and > they have to be installed separately on each node of the clusters, container > images, or local test environments which adds huge complexities from > deployment point of view. In some environments, it requires compiling the > natives from sources which is non-trivial. Also, this approach is platform > dependent; the binary may not work in different platform, so it requires > recompilation. > It requires extra configuration of java.library.path to load the natives, and > it results higher application deployment and maintenance cost for users. > Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which > is JNI-based implementation. It contains native binaries in jar file, and it > can automatically load the native binaries into JVM from jar without any > setup. If a native implementation can not be found for a platform, it can > fallback to pure-java implementation of lz4. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504431628
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -76,6 +64,19 @@ public Lz4Compressor(int directBufferSize, boolean useLz4HC)
{
this.useLz4HC = useLz4HC;
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ if (useLz4HC) {
+lz4Compressor = lz4Factory.highCompressor();
Review comment:
Yeah, sounds good. We can add it later if it is necessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500458&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500458
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:30
Start Date: 14/Oct/20 06:30
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504431628
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -76,6 +64,19 @@ public Lz4Compressor(int directBufferSize, boolean useLz4HC)
{
this.useLz4HC = useLz4HC;
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ if (useLz4HC) {
+lz4Compressor = lz4Factory.highCompressor();
Review comment:
Yeah, sounds good. We can add it later if it is necessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500458)
Time Spent: 3.5h (was: 3h 20m)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504431944
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -76,6 +64,19 @@ public Lz4Compressor(int directBufferSize, boolean useLz4HC)
{
this.useLz4HC = useLz4HC;
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ if (useLz4HC) {
Review comment:
Oh, yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504432504
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -302,11 +303,20 @@ public synchronized long getBytesWritten() {
public synchronized void end() {
}
- private native static void initIDs();
-
- private native int compressBytesDirect();
-
- private native int compressBytesDirectHC();
-
- public native static String getLibraryName();
+ private int compressDirectBuf() {
Review comment:
Ok, currently this PR focuses on turning to use lz4-java. We can
refactor compressor/decompressor classes later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500459&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500459
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:31
Start Date: 14/Oct/20 06:31
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504431944
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -76,6 +64,19 @@ public Lz4Compressor(int directBufferSize, boolean useLz4HC)
{
this.useLz4HC = useLz4HC;
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ if (useLz4HC) {
Review comment:
Oh, yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500459)
Time Spent: 3h 40m (was: 3.5h)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500460&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500460
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:32
Start Date: 14/Oct/20 06:32
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504432504
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -302,11 +303,20 @@ public synchronized long getBytesWritten() {
public synchronized void end() {
}
- private native static void initIDs();
-
- private native int compressBytesDirect();
-
- private native int compressBytesDirectHC();
-
- public native static String getLibraryName();
+ private int compressDirectBuf() {
Review comment:
Ok, currently this PR focuses on turning to use lz4-java. We can
refactor compressor/decompressor classes later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500460)
Time Spent: 3h 50m (was: 3h 40m)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504434296
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -67,6 +55,15 @@
public Lz4Decompressor(int directBufferSize) {
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ lz4Decompressor = lz4Factory.safeDecompressor();
+} catch (Throwable t) {
Review comment:
Hmm, I tried to explicitly catch `ClassNotFoundException`, but the java
compiler complains...
```
exception java.lang.ClassNotFoundException is never thrown in body of
corresponding try statement
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500463&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500463
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:36
Start Date: 14/Oct/20 06:36
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504434296
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -67,6 +55,15 @@
public Lz4Decompressor(int directBufferSize) {
this.directBufferSize = directBufferSize;
+try {
+ LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
+ lz4Decompressor = lz4Factory.safeDecompressor();
+} catch (Throwable t) {
Review comment:
Hmm, I tried to explicitly catch `ClassNotFoundException`, but the java
compiler complains...
```
exception java.lang.ClassNotFoundException is never thrown in body of
corresponding try statement
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500463)
Time Spent: 4h (was: 3h 50m)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500470&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500470
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:51
Start Date: 14/Oct/20 06:51
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441080
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -302,11 +303,20 @@ public synchronized long getBytesWritten() {
public synchronized void end() {
}
- private native static void initIDs();
-
- private native int compressBytesDirect();
-
- private native int compressBytesDirectHC();
-
- public native static String getLibraryName();
+ private int compressDirectBuf() {
+if (uncompressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `uncompressedDirectBuf` for reading
+ uncompressedDirectBuf.limit(uncompressedDirectBufLen).position(0);
+ compressedDirectBuf.clear();
Review comment:
We can remove the `clear` and `limit` calls at the call site.
```java
// Re-initialize the lz4's output direct-buffer
compressedDirectBuf.clear();
compressedDirectBuf.limit(0);
```
But we cannot remove `clear` in `compressDirectBuf` method.
First, lz4-java cannot accept the `compressedDirectBuf` if it is called with
`limit(0)`.
```
[ERROR] Failures:
[ERROR] TestCompressorDecompressor.testCompressorDecompressor:69 Expected
to find 'testCompressorDecompressor error !!!' but got unexpected exception:
net.jpountz.lz4.LZ4Exception: maxDestLen is too small
at
net.jpountz.lz4.LZ4JNICompressor.compress(LZ4JNICompressor.java:69)
at net.jpountz.lz4.LZ4Compressor.compress(LZ4Compressor.java:158)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressDirectBuf(Lz4Compressor.java:310)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compress(Lz4Compressor.java:237)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:286)
at
org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:114)
at
org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:66)
```
Second, even we remove `limit` in the call site, one test still failed:
```
[ERROR]
testCompressorDecompressor(org.apache.hadoop.io.compress.TestCompressorDecompressor)
Time elapsed: 0.539 s <<< FAILURE!
java.lang.AssertionError:
org.apache.hadoop.io.compress.lz4.Lz4Compressor_org.apache.hadoop.io.compress.lz4.Lz4Decompressor-
empty stream compressed output size != 4 expected:<4> but was:<65796>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$3.assertCompression(CompressDecompressTester.java:33
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441080
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##
@@ -302,11 +303,20 @@ public synchronized long getBytesWritten() {
public synchronized void end() {
}
- private native static void initIDs();
-
- private native int compressBytesDirect();
-
- private native int compressBytesDirectHC();
-
- public native static String getLibraryName();
+ private int compressDirectBuf() {
+if (uncompressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `uncompressedDirectBuf` for reading
+ uncompressedDirectBuf.limit(uncompressedDirectBufLen).position(0);
+ compressedDirectBuf.clear();
Review comment:
We can remove the `clear` and `limit` calls at the call site.
```java
// Re-initialize the lz4's output direct-buffer
compressedDirectBuf.clear();
compressedDirectBuf.limit(0);
```
But we cannot remove `clear` in `compressDirectBuf` method.
First, lz4-java cannot accept the `compressedDirectBuf` if it is called with
`limit(0)`.
```
[ERROR] Failures:
[ERROR] TestCompressorDecompressor.testCompressorDecompressor:69 Expected
to find 'testCompressorDecompressor error !!!' but got unexpected exception:
net.jpountz.lz4.LZ4Exception: maxDestLen is too small
at
net.jpountz.lz4.LZ4JNICompressor.compress(LZ4JNICompressor.java:69)
at net.jpountz.lz4.LZ4Compressor.compress(LZ4Compressor.java:158)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressDirectBuf(Lz4Compressor.java:310)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compress(Lz4Compressor.java:237)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:286)
at
org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:114)
at
org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:66)
```
Second, even we remove `limit` in the call site, one test still failed:
```
[ERROR]
testCompressorDecompressor(org.apache.hadoop.io.compress.TestCompressorDecompressor)
Time elapsed: 0.539 s <<< FAILURE!
java.lang.AssertionError:
org.apache.hadoop.io.compress.lz4.Lz4Compressor_org.apache.hadoop.io.compress.lz4.Lz4Decompressor-
empty stream compressed output size != 4 expected:<4> but was:<65796>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$3.assertCompression(CompressDecompressTester.java:334)
at
org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:114)
at
org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:66)
```
This is an automated message from the Apache Git Service.
To respond to the message, please lo
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441522
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -272,7 +269,19 @@ public synchronized void end() {
// do nothing
}
- private native static void initIDs();
-
- private native int decompressBytesDirect();
+ private int decompressDirectBuf() {
+if (compressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `compressedDirectBuf` for reading
Review comment:
ok, removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441816
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -272,7 +269,19 @@ public synchronized void end() {
// do nothing
}
- private native static void initIDs();
-
- private native int decompressBytesDirect();
+ private int decompressDirectBuf() {
+if (compressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `compressedDirectBuf` for reading
+ compressedDirectBuf.limit(compressedDirectBufLen).position(0);
+ lz4Decompressor.decompress((ByteBuffer) compressedDirectBuf,
+ (ByteBuffer) uncompressedDirectBuf);
+ compressedDirectBufLen = 0;
+ compressedDirectBuf.limit(compressedDirectBuf.capacity()).position(0);
Review comment:
sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500472&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500472
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:53
Start Date: 14/Oct/20 06:53
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441816
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -272,7 +269,19 @@ public synchronized void end() {
// do nothing
}
- private native static void initIDs();
-
- private native int decompressBytesDirect();
+ private int decompressDirectBuf() {
+if (compressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `compressedDirectBuf` for reading
+ compressedDirectBuf.limit(compressedDirectBufLen).position(0);
+ lz4Decompressor.decompress((ByteBuffer) compressedDirectBuf,
+ (ByteBuffer) uncompressedDirectBuf);
+ compressedDirectBufLen = 0;
+ compressedDirectBuf.limit(compressedDirectBuf.capacity()).position(0);
Review comment:
sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500472)
Time Spent: 4.5h (was: 4h 20m)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[ https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500473 ] ASF GitHub Bot logged work on HADOOP-17292: --- Author: ASF GitHub Bot Created on: 14/Oct/20 06:54 Start Date: 14/Oct/20 06:54 Worklog Time Spent: 10m Work Description: viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504442381 ## File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml ## @@ -71,6 +71,11 @@ assertj-core test + Review comment: One test in `hadoop-mapreduce-client-nativetask` uses `Lz4Codec`. If we don't add `lz4-java` as test dependency, because it is `provided` scope in `hadoop-common` now, we will get class not found exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500473) Time Spent: 4h 40m (was: 4.5h) > Using lz4-java in Lz4Codec > -- > > Key: HADOOP-17292 > URL: https://issues.apache.org/jira/browse/HADOOP-17292 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > In Hadoop, we use native libs for lz4 codec which has several disadvantages: > It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and > they have to be installed separately on each node of the clusters, container > images, or local test environments which adds huge complexities from > deployment point of view. In some environments, it requires compiling the > natives from sources which is non-trivial. Also, this approach is platform > dependent; the binary may not work in different platform, so it requires > recompilation. > It requires extra configuration of java.library.path to load the natives, and > it results higher application deployment and maintenance cost for users. > Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which > is JNI-based implementation. It contains native binaries in jar file, and it > can automatically load the native binaries into JVM from jar without any > setup. If a native implementation can not be found for a platform, it can > fallback to pure-java implementation of lz4. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17292) Using lz4-java in Lz4Codec
[
https://issues.apache.org/jira/browse/HADOOP-17292?focusedWorklogId=500471&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500471
]
ASF GitHub Bot logged work on HADOOP-17292:
---
Author: ASF GitHub Bot
Created on: 14/Oct/20 06:53
Start Date: 14/Oct/20 06:53
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441522
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
##
@@ -272,7 +269,19 @@ public synchronized void end() {
// do nothing
}
- private native static void initIDs();
-
- private native int decompressBytesDirect();
+ private int decompressDirectBuf() {
+if (compressedDirectBufLen == 0) {
+ return 0;
+} else {
+ // Set the position and limit of `compressedDirectBuf` for reading
Review comment:
ok, removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 500471)
Time Spent: 4h 20m (was: 4h 10m)
> Using lz4-java in Lz4Codec
> --
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Affects Versions: 3.3.0
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #2350: HADOOP-17292. Using lz4-java in Lz4Codec
viirya commented on a change in pull request #2350: URL: https://github.com/apache/hadoop/pull/2350#discussion_r504442381 ## File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml ## @@ -71,6 +71,11 @@ assertj-core test + Review comment: One test in `hadoop-mapreduce-client-nativetask` uses `Lz4Codec`. If we don't add `lz4-java` as test dependency, because it is `provided` scope in `hadoop-common` now, we will get class not found exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
