[GitHub] [hadoop] hadoop-yetus commented on pull request #2665: YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread
hadoop-yetus commented on pull request #2665:
URL: https://github.com/apache/hadoop/pull/2665#issuecomment-770687212
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 49s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 56s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 1m 53s |
[/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2665/1/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant findbugs warnings. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 51s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 56s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 2m 1s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 95m 45s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2665/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 177m 2s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2665/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2665 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a57ab5a4d467 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1a205cc3adf |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
/us
[jira] [Commented] (HADOOP-17424) Replace HTrace with No-Op tracer
[
https://issues.apache.org/jira/browse/HADOOP-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276195#comment-17276195
]
Siyao Meng commented on HADOOP-17424:
-
Thanks [~iwasakims]. Indeed the removal of {{TraceAdminProtocol}} (among other
interfaces used by HTrace) might cause compatibility issues and thus this
should only go in 3.4.0.
> Replace HTrace with No-Op tracer
>
>
> Key: HADOOP-17424
> URL: https://issues.apache.org/jira/browse/HADOOP-17424
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
> Remove HTrace dependency as it is depending on old jackson jars. Use a no-op
> tracer for now to eliminate potential security issues.
> The plan is to move part of the code in
> [PR#1846|https://github.com/apache/hadoop/pull/1846] out here for faster
> review.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HeartSaVioR opened a new pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
HeartSaVioR opened a new pull request #2667: URL: https://github.com/apache/hadoop/pull/2667 NOTE: WIP. DO-NOT-MERGE. No meaningful tests have been added, as I have no idea where I can add it, and how s3a has been tested with integration test manner. (Tests in TestS3ABlockOutputStream only check simple things with mocking everything, so can't do some write/upload test with it.) I got some review comments from @steveloughran in my commit, and will reflect these review comments. Once it's done I'll remove WIP. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-16906:
Labels: pull-request-available (was: )
> Add some Abortable.abort() interface for streams etc which can be terminated
>
>
> Key: HADOOP-16906
> URL: https://issues.apache.org/jira/browse/HADOOP-16906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some IO we want to be able to abort rather than close cleanly, especially if
> the inner stream is an HTTP connection which itself supports some abort()
> method. For example: uploads to an object where we want to cancel the upload
> without close() making an incomplete write visible.
> Proposed: Add a generic interface which things like streams can implement
> {code}
> AbortableIO {
> public void abortIO() throws IOE;
> }
> {code}
> +do for s3a output stream. I wouldn't do this a passthrough on
> FSDataOutputStream because we need to consider what expectations callers have
> of an operation being "aborted"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545210
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 09:40
Start Date: 01/Feb/21 09:40
Worklog Time Spent: 10m
Work Description: HeartSaVioR opened a new pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667
NOTE: WIP. DO-NOT-MERGE.
No meaningful tests have been added, as I have no idea where I can add it,
and how s3a has been tested
with integration test manner. (Tests in TestS3ABlockOutputStream only check
simple things with mocking
everything, so can't do some write/upload test with it.)
I got some review comments from @steveloughran in my commit, and will
reflect these review comments.
Once it's done I'll remove WIP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545210)
Remaining Estimate: 0h
Time Spent: 10m
> Add some Abortable.abort() interface for streams etc which can be terminated
>
>
> Key: HADOOP-16906
> URL: https://issues.apache.org/jira/browse/HADOOP-16906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some IO we want to be able to abort rather than close cleanly, especially if
> the inner stream is an HTTP connection which itself supports some abort()
> method. For example: uploads to an object where we want to cancel the upload
> without close() making an incomplete write visible.
> Proposed: Add a generic interface which things like streams can implement
> {code}
> AbortableIO {
> public void abortIO() throws IOE;
> }
> {code}
> +do for s3a output stream. I wouldn't do this a passthrough on
> FSDataOutputStream because we need to consider what expectations callers have
> of an operation being "aborted"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HeartSaVioR commented on pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
HeartSaVioR commented on pull request #2667: URL: https://github.com/apache/hadoop/pull/2667#issuecomment-770721288 I'll migrate @steveloughran 's review comments to the diff in PR. It doesn't seem to be done automatically - these comments have been shown as "normal review comments". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545211&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545211
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 09:42
Start Date: 01/Feb/21 09:42
Worklog Time Spent: 10m
Work Description: HeartSaVioR commented on pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#issuecomment-770721288
I'll migrate @steveloughran 's review comments to the diff in PR. It doesn't
seem to be done automatically - these comments have been shown as "normal
review comments".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545211)
Time Spent: 20m (was: 10m)
> Add some Abortable.abort() interface for streams etc which can be terminated
>
>
> Key: HADOOP-16906
> URL: https://issues.apache.org/jira/browse/HADOOP-16906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Some IO we want to be able to abort rather than close cleanly, especially if
> the inner stream is an HTTP connection which itself supports some abort()
> method. For example: uploads to an object where we want to cancel the upload
> without close() making an incomplete write visible.
> Proposed: Add a generic interface which things like streams can implement
> {code}
> AbortableIO {
> public void abortIO() throws IOE;
> }
> {code}
> +do for s3a output stream. I wouldn't do this a passthrough on
> FSDataOutputStream because we need to consider what expectations callers have
> of an operation being "aborted"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HeartSaVioR commented on a change in pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
HeartSaVioR commented on a change in pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#discussion_r567685195
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46507150
> the new "cloud ready" API calls always return a CompletableFuture, to
emphasise that the op may take time and to allow the caller to do something
while waiting. Would we want to do this here? I'm not convinced it is
appropriate. Instead we say
>
> 1. call must guarantee that after this is invoked,. close() will not
materialize the file at its final path
> 2. it may communicate with the store to cancel an operation; which may
retry. Errors will be stored.
> 3. there may still/also be async IO to the store after the call returns,
but this must maintain the requirement "not visible"
> 4. And close() may do some IO to cancel
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
+if (closed.getAndSet(true)) {
+ // already closed
+ LOG.debug("Ignoring abort() as stream is already closed");
+ return;
+}
+
+S3ADataBlocks.DataBlock block = getActiveBlock();
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506593
> I think we are going to have to worry about this a bit more, because we
may have queued >1 block for upload in a separate thread. They'll maybe need
interruption, or at least, when they finish, see if they should immediately
cancel the upload. This won't make any difference in the semantics of abort()
(the final upload has been killed), I just don't want to run up any bills.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
+if (closed.getAndSet(true)) {
+ // already closed
+ LOG.debug("Ignoring abort() as stream is already closed");
+ return;
+}
+
+S3ADataBlocks.DataBlock block = getActiveBlock();
+try {
+ if (multiPartUpload != null) {
+multiPartUpload.abort();
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506765
> ok, don't worry so much about my prev comment. That cancels all the
outstanding futures.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlockOutputStream.java
##
@@ -82,4 +82,22 @@ public void testWriteOperationHelperPartLimits() throws
Throwable {
() -> woh.newUploadPartRequest(key,
"uploadId", 5, 1024, inputStream, null, 0L));
}
+
+ @Test
+ public void testStreamClosedAfterAbort() throws Exception {
+stream.abort();
+
+// This verification replaces testing various operations after calling
abort:
+// after calling abort, stream is closed like calling close().
+intercept(IOException.class, () -> stream.checkOpen());
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46507220
> should also verify that stream.write() raises an IOE. We could raise a
subclass of IOE to indicate this was a checkOpen failure for a stricter test
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlockOutputStream.java
##
@@ -82,4 +82,22 @@ public void testWriteOperationHelperPartLimits() throws
Throwable {
() -> woh.newUploadPartRequest(key,
"uploadId", 5, 1024, inputStream, null, 0L));
}
+
+ @Test
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506646
> tests are good. We will need to do an ITest too, which can be done in
ITestS3ABlockOutputArray
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issu
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545216
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 09:48
Start Date: 01/Feb/21 09:48
Worklog Time Spent: 10m
Work Description: HeartSaVioR commented on a change in pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#discussion_r567685195
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46507150
> the new "cloud ready" API calls always return a CompletableFuture, to
emphasise that the op may take time and to allow the caller to do something
while waiting. Would we want to do this here? I'm not convinced it is
appropriate. Instead we say
>
> 1. call must guarantee that after this is invoked,. close() will not
materialize the file at its final path
> 2. it may communicate with the store to cancel an operation; which may
retry. Errors will be stored.
> 3. there may still/also be async IO to the store after the call returns,
but this must maintain the requirement "not visible"
> 4. And close() may do some IO to cancel
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
+if (closed.getAndSet(true)) {
+ // already closed
+ LOG.debug("Ignoring abort() as stream is already closed");
+ return;
+}
+
+S3ADataBlocks.DataBlock block = getActiveBlock();
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506593
> I think we are going to have to worry about this a bit more, because we
may have queued >1 block for upload in a separate thread. They'll maybe need
interruption, or at least, when they finish, see if they should immediately
cancel the upload. This won't make any difference in the semantics of abort()
(the final upload has been killed), I just don't want to run up any bills.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
##
@@ -551,6 +556,24 @@ public IOStatistics getIOStatistics() {
return iostatistics;
}
+ @Override
+ public void abort() {
+if (closed.getAndSet(true)) {
+ // already closed
+ LOG.debug("Ignoring abort() as stream is already closed");
+ return;
+}
+
+S3ADataBlocks.DataBlock block = getActiveBlock();
+try {
+ if (multiPartUpload != null) {
+multiPartUpload.abort();
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506765
> ok, don't worry so much about my prev comment. That cancels all the
outstanding futures.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlockOutputStream.java
##
@@ -82,4 +82,22 @@ public void testWriteOperationHelperPartLimits() throws
Throwable {
() -> woh.newUploadPartRequest(key,
"uploadId", 5, 1024, inputStream, null, 0L));
}
+
+ @Test
+ public void testStreamClosedAfterAbort() throws Exception {
+stream.abort();
+
+// This verification replaces testing various operations after calling
abort:
+// after calling abort, stream is closed like calling close().
+intercept(IOException.class, () -> stream.checkOpen());
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46507220
> should also verify that stream.write() raises an IOE. We could raise a
subclass of IOE to indicate this was a checkOpen failure for a stricter test
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlockOutputStream.java
##
@@ -82,4 +82,22 @@ public void testWriteOperationHelperPartLimits() throws
Throwable {
() -> woh.newUploadPartRequest(key,
"uploadId", 5, 1024, inputStream, null, 0L));
}
+
+ @Test
Review comment:
Migrating comment in
https://github.com/HeartSaVioR/hadoop/commit/63c5588d46d422a66ec620ae310f9a860db890b3#r46506646
> tests are good. We will need to do an ITest too, which can be done in
ITestS3ABlockOutputArray
This
[GitHub] [hadoop] aajisaka commented on pull request #2647: HADOOP-17354. Move Jenkinsfile outside of the root directory.
aajisaka commented on pull request #2647: URL: https://github.com/apache/hadoop/pull/2647#issuecomment-770726130 Thank you @ayushtkn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17354) Move Jenkinsfile outside of the root directory
[ https://issues.apache.org/jira/browse/HADOOP-17354?focusedWorklogId=545220&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545220 ] ASF GitHub Bot logged work on HADOOP-17354: --- Author: ASF GitHub Bot Created on: 01/Feb/21 09:50 Start Date: 01/Feb/21 09:50 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #2647: URL: https://github.com/apache/hadoop/pull/2647#issuecomment-770726130 Thank you @ayushtkn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545220) Time Spent: 50m (was: 40m) > Move Jenkinsfile outside of the root directory > -- > > Key: HADOOP-17354 > URL: https://issues.apache.org/jira/browse/HADOOP-17354 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The Jenkinsfile is placed under the project root directory, so when the > Jenkinsfile is changed, all the Hadoop unit tests will run and it wastes a > lot of time and resources. Let's move the file outside of the root directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2647: HADOOP-17354. Move Jenkinsfile outside of the root directory.
aajisaka merged pull request #2647: URL: https://github.com/apache/hadoop/pull/2647 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17354) Move Jenkinsfile outside of the root directory
[ https://issues.apache.org/jira/browse/HADOOP-17354?focusedWorklogId=545221&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545221 ] ASF GitHub Bot logged work on HADOOP-17354: --- Author: ASF GitHub Bot Created on: 01/Feb/21 09:51 Start Date: 01/Feb/21 09:51 Worklog Time Spent: 10m Work Description: aajisaka merged pull request #2647: URL: https://github.com/apache/hadoop/pull/2647 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545221) Time Spent: 1h (was: 50m) > Move Jenkinsfile outside of the root directory > -- > > Key: HADOOP-17354 > URL: https://issues.apache.org/jira/browse/HADOOP-17354 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The Jenkinsfile is placed under the project root directory, so when the > Jenkinsfile is changed, all the Hadoop unit tests will run and it wastes a > lot of time and resources. Let's move the file outside of the root directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17354) Move Jenkinsfile outside of the root directory
[ https://issues.apache.org/jira/browse/HADOOP-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17354. Fix Version/s: 3.2.3 2.10.2 3.1.5 3.4.0 3.3.1 Resolution: Fixed Merged the PR into all the active branches, and I've update the config of hadoop-multibranch job. > Move Jenkinsfile outside of the root directory > -- > > Key: HADOOP-17354 > URL: https://issues.apache.org/jira/browse/HADOOP-17354 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Time Spent: 1h > Remaining Estimate: 0h > > The Jenkinsfile is placed under the project root directory, so when the > Jenkinsfile is changed, all the Hadoop unit tests will run and it wastes a > lot of time and resources. Let's move the file outside of the root directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17354) Move Jenkinsfile outside of the root directory
[ https://issues.apache.org/jira/browse/HADOOP-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276216#comment-17276216 ] Akira Ajisaka edited comment on HADOOP-17354 at 2/1/21, 9:56 AM: - Merged the PR into all the active branches. Also, I've updated the config of hadoop-multibranch job. was (Author: ajisakaa): Merged the PR into all the active branches, and I've update the config of hadoop-multibranch job. > Move Jenkinsfile outside of the root directory > -- > > Key: HADOOP-17354 > URL: https://issues.apache.org/jira/browse/HADOOP-17354 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Time Spent: 1h > Remaining Estimate: 0h > > The Jenkinsfile is placed under the project root directory, so when the > Jenkinsfile is changed, all the Hadoop unit tests will run and it wastes a > lot of time and resources. Let's move the file outside of the root directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #2652: HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf
jojochuang merged pull request #2652: URL: https://github.com/apache/hadoop/pull/2652 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on a change in pull request #2646: HADOOP-17038 Support disabling buffered reads in ABFS positional reads.
surendralilhore commented on a change in pull request #2646: URL: https://github.com/apache/hadoop/pull/2646#discussion_r567700230 ## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ## @@ -812,6 +812,13 @@ aheads. Specify the value in bytes. The value should be between 16384 to 104857600 both inclusive (16 KB to 100 MB). The default value will be 4194304 (4 MB). +`fs.azure.buffered.pread.disable`: By default the positional read API will do a Review comment: We can clearly mentioned here that, it is not a property and it's used as a option in FutureDataInputStreamBuilder. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[ https://issues.apache.org/jira/browse/HADOOP-17038?focusedWorklogId=545228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545228 ] ASF GitHub Bot logged work on HADOOP-17038: --- Author: ASF GitHub Bot Created on: 01/Feb/21 10:07 Start Date: 01/Feb/21 10:07 Worklog Time Spent: 10m Work Description: surendralilhore commented on a change in pull request #2646: URL: https://github.com/apache/hadoop/pull/2646#discussion_r567700230 ## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ## @@ -812,6 +812,13 @@ aheads. Specify the value in bytes. The value should be between 16384 to 104857600 both inclusive (16 KB to 100 MB). The default value will be 4194304 (4 MB). +`fs.azure.buffered.pread.disable`: By default the positional read API will do a Review comment: We can clearly mentioned here that, it is not a property and it's used as a option in FutureDataInputStreamBuilder. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545228) Time Spent: 2.5h (was: 2h 20m) > Support disabling buffered reads in ABFS positional reads > - > > Key: HADOOP-17038 > URL: https://issues.apache.org/jira/browse/HADOOP-17038 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Labels: HBase, abfsactive, pull-request-available > Attachments: HBase Perf Test Report.xlsx, screenshot-1.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > Right now it will do a seek to the position , read and then seek back to the > old position. (As per the impl in the super class) > In HBase kind of workloads we rely mostly on short preads. (like 64 KB size > by default). So would be ideal to support a pure pos read API which will not > even keep the data in a buffer but will only read the required data as what > is asked for by the caller. (Not reading ahead more data as per the read size > config) > Allow an optional boolean config to be specified while opening file for read > using which buffered pread can be disabled. > FutureDataInputStreamBuilder openFile(Path path) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x
hadoop-yetus commented on pull request #763: URL: https://github.com/apache/hadoop/pull/763#issuecomment-770752229 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 5s | | https://github.com/apache/hadoop/pull/763 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/763 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-763/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?focusedWorklogId=545243&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545243 ] ASF GitHub Bot logged work on HADOOP-15984: --- Author: ASF GitHub Bot Created on: 01/Feb/21 10:31 Start Date: 01/Feb/21 10:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #763: URL: https://github.com/apache/hadoop/pull/763#issuecomment-770752229 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 5s | | https://github.com/apache/hadoop/pull/763 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/763 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-763/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545243) Time Spent: 1h 20m (was: 1h 10m) > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on a change in pull request #2646: HADOOP-17038 Support disabling buffered reads in ABFS positional reads.
anoopsjohn commented on a change in pull request #2646: URL: https://github.com/apache/hadoop/pull/2646#discussion_r567720601 ## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ## @@ -812,6 +812,13 @@ aheads. Specify the value in bytes. The value should be between 16384 to 104857600 both inclusive (16 KB to 100 MB). The default value will be 4194304 (4 MB). +`fs.azure.buffered.pread.disable`: By default the positional read API will do a Review comment: Thanks.. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[ https://issues.apache.org/jira/browse/HADOOP-17038?focusedWorklogId=545249&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545249 ] ASF GitHub Bot logged work on HADOOP-17038: --- Author: ASF GitHub Bot Created on: 01/Feb/21 10:38 Start Date: 01/Feb/21 10:38 Worklog Time Spent: 10m Work Description: anoopsjohn commented on a change in pull request #2646: URL: https://github.com/apache/hadoop/pull/2646#discussion_r567720601 ## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ## @@ -812,6 +812,13 @@ aheads. Specify the value in bytes. The value should be between 16384 to 104857600 both inclusive (16 KB to 100 MB). The default value will be 4194304 (4 MB). +`fs.azure.buffered.pread.disable`: By default the positional read API will do a Review comment: Thanks.. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545249) Time Spent: 2h 40m (was: 2.5h) > Support disabling buffered reads in ABFS positional reads > - > > Key: HADOOP-17038 > URL: https://issues.apache.org/jira/browse/HADOOP-17038 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Labels: HBase, abfsactive, pull-request-available > Attachments: HBase Perf Test Report.xlsx, screenshot-1.png > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Right now it will do a seek to the position , read and then seek back to the > old position. (As per the impl in the super class) > In HBase kind of workloads we rely mostly on short preads. (like 64 KB size > by default). So would be ideal to support a pure pos read API which will not > even keep the data in a buffer but will only read the required data as what > is asked for by the caller. (Not reading ahead more data as per the read size > config) > Allow an optional boolean config to be specified while opening file for read > using which buffered pread can be disabled. > FutureDataInputStreamBuilder openFile(Path path) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2646: HADOOP-17038 Support disabling buffered reads in ABFS positional reads.
hadoop-yetus commented on pull request #2646:
URL: https://github.com/apache/hadoop/pull/2646#issuecomment-770806203
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 37s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 0m 56s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 8s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 83m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 62bbb94de8fc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 115623a6ee4 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/testReport/ |
| Max. process+thread count | 510 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
-
[jira] [Work logged] (HADOOP-17038) Support disabling buffered reads in ABFS positional reads
[
https://issues.apache.org/jira/browse/HADOOP-17038?focusedWorklogId=545287&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545287
]
ASF GitHub Bot logged work on HADOOP-17038:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 12:02
Start Date: 01/Feb/21 12:02
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2646:
URL: https://github.com/apache/hadoop/pull/2646#issuecomment-770806203
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 37s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 0m 56s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 8s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 83m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 62bbb94de8fc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 115623a6ee4 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2646/2/testReport/ |
| Max. proce
[GitHub] [hadoop] ravwojdyla commented on a change in pull request #2638: HADOOP-17402. Add GCS config to the core-site
ravwojdyla commented on a change in pull request #2638:
URL: https://github.com/apache/hadoop/pull/2638#discussion_r567774217
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
##
@@ -142,6 +141,7 @@ public void initializeMemberVariables() {
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.webhdfs.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.wasb.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.swift.impl");
+xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.gs.impl");
Review comment:
@medb I can certainly change this, but it's already not sorted due to
`swift`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17402) Add GCS FS impl reference to core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-17402?focusedWorklogId=545288&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545288
]
ASF GitHub Bot logged work on HADOOP-17402:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 12:06
Start Date: 01/Feb/21 12:06
Worklog Time Spent: 10m
Work Description: ravwojdyla commented on a change in pull request #2638:
URL: https://github.com/apache/hadoop/pull/2638#discussion_r567774217
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
##
@@ -142,6 +141,7 @@ public void initializeMemberVariables() {
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.webhdfs.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.wasb.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.swift.impl");
+xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.gs.impl");
Review comment:
@medb I can certainly change this, but it's already not sorted due to
`swift`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545288)
Time Spent: 2h (was: 1h 50m)
> Add GCS FS impl reference to core-default.xml
> -
>
> Key: HADOOP-17402
> URL: https://issues.apache.org/jira/browse/HADOOP-17402
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Rafal Wojdyla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Akin to current S3 default configuration add GCS configuration, specifically
> to declare the GCS implementation. [GCS
> connector|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage].
> Has this not been done since the GCS connector is not part of the hadoop/ASF
> codebase, or is there any other blocker?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2659: HDFS-15801. Backport HDFS-14582 to branch-2.10 (Failed to start DN wi…
hadoop-yetus commented on pull request #2659: URL: https://github.com/apache/hadoop/pull/2659#issuecomment-770815635 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 12m 57s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 26s | branch-2.10 passed | | +1 :green_heart: | compile | 1m 0s | branch-2.10 passed | | +1 :green_heart: | checkstyle | 0m 37s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 1m 14s | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 16s | branch-2.10 passed | | +0 :ok: | spotbugs | 3m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 58s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 4s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | checkstyle | 0m 30s | the patch passed | | +1 :green_heart: | mvnsite | 1m 6s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 1m 10s | the patch passed | | +1 :green_heart: | findbugs | 3m 4s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 62m 46s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 113m 51s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys | | | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2659/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2659 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 969b20a1d28d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / dded428 | | Default Java | Oracle Corporation-1.7.0_95-b00 | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2659/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2659/2/testReport/ | | Max. process+thread count | 2410 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2659/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut opened a new pull request #2668: HDFS-15808. Add metrics for FSNamesystem read/write lock warnings
tomscut opened a new pull request #2668: URL: https://github.com/apache/hadoop/pull/2668 JIRA: https://issues.apache.org/jira/browse/HDFS-15808 To monitor how often read/write locks exceed thresholds, we can add two metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2666: HDFS-15807. RefreshVolume fails when replacing DISK/ARCHIVE vol on same mount.
hadoop-yetus commented on pull request #2666:
URL: https://github.com/apache/hadoop/pull/2666#issuecomment-770838565
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 43s | | trunk passed |
| +1 :green_heart: | compile | 1m 41s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 7s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 3m 19s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 3m 13s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 211m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2666/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 307m 11s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2666/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2666 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0b2ced947406 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1a205cc3adf |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2666/1/testReport/ |
| Max. process+thread count | 2048 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2666/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
[GitHub] [hadoop] hadoop-yetus commented on pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
hadoop-yetus commented on pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#issuecomment-770846140
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 29s | | trunk passed |
| +1 :green_heart: | compile | 24m 14s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 12s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 21m 32s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 32s | | the patch passed |
| +1 :green_heart: | compile | 19m 2s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 19m 2s | | the patch passed |
| -0 :warning: | checkstyle | 3m 51s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| -1 :x: | findbugs | 1m 21s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total
(was 0) |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 18s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 53s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 209m 22s | | |
| Reason | Tests |
|---:|:--|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.multiPartUpload; locked 50% of
time Unsynchronized access at S3ABlockOutputStream.java:50% of time
Unsynchronized access at S3ABlockOutputStream.java:[line 569] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2667 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 45196ae1217f 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1a205cc3adf |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545320
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 13:11
Start Date: 01/Feb/21 13:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#issuecomment-770846140
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 29s | | trunk passed |
| +1 :green_heart: | compile | 24m 14s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 12s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 21m 32s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 32s | | the patch passed |
| +1 :green_heart: | compile | 19m 2s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 19m 2s | | the patch passed |
| -0 :warning: | checkstyle | 3m 51s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| -1 :x: | findbugs | 1m 21s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total
(was 0) |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 18s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 53s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 209m 22s | | |
| Reason | Tests |
|---:|:--|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.multiPartUpload; locked 50% of
time Unsynchronized access at S3ABlockOutputStream.java:50% of time
Unsynchronized access at S3ABlockOutputStream.java:[line 569] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2667 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit
[GitHub] [hadoop] steveloughran commented on pull request #1839: HADOOP-16848. Refactoring: initial layering
steveloughran commented on pull request #1839: URL: https://github.com/apache/hadoop/pull/1839#issuecomment-770941645 * Not Tested* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16848) add initial S3A layering + async init
[ https://issues.apache.org/jira/browse/HADOOP-16848?focusedWorklogId=545403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545403 ] ASF GitHub Bot logged work on HADOOP-16848: --- Author: ASF GitHub Bot Created on: 01/Feb/21 15:30 Start Date: 01/Feb/21 15:30 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #1839: URL: https://github.com/apache/hadoop/pull/1839#issuecomment-770941645 * Not Tested* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545403) Time Spent: 1h 20m (was: 1h 10m) > add initial S3A layering + async init > - > > Key: HADOOP-16848 > URL: https://issues.apache.org/jira/browse/HADOOP-16848 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Split the S3A code into layers > * S3AFileSystem > * S3AStore + Impl > * RawS3A + Impl > S3AFS will create the others and start in order: DelegationTokens, RawS3A, > S3AStore, Metastore > this will involve wrapping all access of DTs, s3client, Metastore to block > until that layer is complete, or raise an exception if instantiation of > it/predecessor failed. > New layers will all be subclasses of Service, split into Interface and Impl, > so we can manage the init/start/stop lifecycle with existing code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17509) Parallelize building of Protocol buffers
Gautham Banasandra created HADOOP-17509: --- Summary: Parallelize building of Protocol buffers Key: HADOOP-17509 URL: https://issues.apache.org/jira/browse/HADOOP-17509 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.3.0 Reporter: Gautham Banasandra Assignee: Gautham Banasandra Need to use make -j$(nproc) to parallelize building of Protocol buffers dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17509) Parallelize building of dependencies
[ https://issues.apache.org/jira/browse/HADOOP-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautham Banasandra updated HADOOP-17509: Summary: Parallelize building of dependencies (was: Parallelize building of Protocol buffers) > Parallelize building of dependencies > > > Key: HADOOP-17509 > URL: https://issues.apache.org/jira/browse/HADOOP-17509 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > > Need to use make -j$(nproc) to parallelize building of Protocol buffers > dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17509) Parallelize building of dependencies
[ https://issues.apache.org/jira/browse/HADOOP-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautham Banasandra updated HADOOP-17509: Description: Need to use make -j$(nproc) to parallelize building of Protocol buffers and Intel ISA - L dependency. (was: Need to use make -j$(nproc) to parallelize building of Protocol buffers dependency.) > Parallelize building of dependencies > > > Key: HADOOP-17509 > URL: https://issues.apache.org/jira/browse/HADOOP-17509 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > > Need to use make -j$(nproc) to parallelize building of Protocol buffers and > Intel ISA - L dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request #2669: HADOOP-17509. Parallilize building of deps
GauthamBanasandra opened a new pull request #2669: URL: https://github.com/apache/hadoop/pull/2669 * Added -j option to parallelize the building of Protocol buffers and Intel ISA-L dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17509) Parallelize building of dependencies
[ https://issues.apache.org/jira/browse/HADOOP-17509?focusedWorklogId=545407&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545407 ] ASF GitHub Bot logged work on HADOOP-17509: --- Author: ASF GitHub Bot Created on: 01/Feb/21 15:47 Start Date: 01/Feb/21 15:47 Worklog Time Spent: 10m Work Description: GauthamBanasandra opened a new pull request #2669: URL: https://github.com/apache/hadoop/pull/2669 * Added -j option to parallelize the building of Protocol buffers and Intel ISA-L dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545407) Remaining Estimate: 0h Time Spent: 10m > Parallelize building of dependencies > > > Key: HADOOP-17509 > URL: https://issues.apache.org/jira/browse/HADOOP-17509 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Need to use make -j$(nproc) to parallelize building of Protocol buffers and > Intel ISA - L dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17509) Parallelize building of dependencies
[ https://issues.apache.org/jira/browse/HADOOP-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17509: Labels: pull-request-available (was: ) > Parallelize building of dependencies > > > Key: HADOOP-17509 > URL: https://issues.apache.org/jira/browse/HADOOP-17509 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Need to use make -j$(nproc) to parallelize building of Protocol buffers and > Intel ISA - L dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #2664: HADOOP-17508. Simplify deps install instructions
GauthamBanasandra commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r567927981 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src Review comment: Thanks for the catch @smengcl. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17508) Simplify dependency installation instructions
[ https://issues.apache.org/jira/browse/HADOOP-17508?focusedWorklogId=545409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545409 ] ASF GitHub Bot logged work on HADOOP-17508: --- Author: ASF GitHub Bot Created on: 01/Feb/21 15:48 Start Date: 01/Feb/21 15:48 Worklog Time Spent: 10m Work Description: GauthamBanasandra commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r567928611 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src + $ ./configure + $ make -j$(nproc) Review comment: Thanks 😁. Well, now that you mentioned it, could you please review - https://github.com/apache/hadoop/pull/2669 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545409) Time Spent: 1h (was: 50m) > Simplify dependency installation instructions > - > > Key: HADOOP-17508 > URL: https://issues.apache.org/jira/browse/HADOOP-17508 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Trivial > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The instructions for installing Protocol buffers and Boost seem to just be > copied from Dockerfile and isn't friendly to read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #2664: HADOOP-17508. Simplify deps install instructions
GauthamBanasandra commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r567928611 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src + $ ./configure + $ make -j$(nproc) Review comment: Thanks 😁. Well, now that you mentioned it, could you please review - https://github.com/apache/hadoop/pull/2669 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17508) Simplify dependency installation instructions
[ https://issues.apache.org/jira/browse/HADOOP-17508?focusedWorklogId=545408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545408 ] ASF GitHub Bot logged work on HADOOP-17508: --- Author: ASF GitHub Bot Created on: 01/Feb/21 15:48 Start Date: 01/Feb/21 15:48 Worklog Time Spent: 10m Work Description: GauthamBanasandra commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r567927981 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src Review comment: Thanks for the catch @smengcl. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545408) Time Spent: 50m (was: 40m) > Simplify dependency installation instructions > - > > Key: HADOOP-17508 > URL: https://issues.apache.org/jira/browse/HADOOP-17508 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Trivial > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The instructions for installing Protocol buffers and Boost seem to just be > copied from Dockerfile and isn't friendly to read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2669: HADOOP-17509. Parallilize building of deps
hadoop-yetus commented on pull request #2669: URL: https://github.com/apache/hadoop/pull/2669#issuecomment-77094 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17509) Parallelize building of dependencies
[ https://issues.apache.org/jira/browse/HADOOP-17509?focusedWorklogId=545410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545410 ] ASF GitHub Bot logged work on HADOOP-17509: --- Author: ASF GitHub Bot Created on: 01/Feb/21 15:50 Start Date: 01/Feb/21 15:50 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2669: URL: https://github.com/apache/hadoop/pull/2669#issuecomment-77094 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545410) Time Spent: 20m (was: 10m) > Parallelize building of dependencies > > > Key: HADOOP-17509 > URL: https://issues.apache.org/jira/browse/HADOOP-17509 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Need to use make -j$(nproc) to parallelize building of Protocol buffers and > Intel ISA - L dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2595: HDFS-15619 Add snapshot related metrics
bshashikant commented on a change in pull request #2595:
URL: https://github.com/apache/hadoop/pull/2595#discussion_r567939782
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
##
@@ -459,4 +472,23 @@ public void addEditLogTailInterval(long elapsed) {
q.add(elapsed);
}
}
+ public void incrNumSnapshotDelete() {
+numSnapshotTotalDeleteOp.incr();
Review comment:
Its already an existing metric...Let's remove it.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##
@@ -519,6 +520,14 @@ public void deleteSnapshot(final INodesInPath iip, final
String snapshotName,
final List xattrs = Lists.newArrayListWithCapacity(1);
xattrs.add(snapshotXAttr);
+// Check if snapshot is already marked as deleted.
+// If already marked deleted, metric for out of order snapshot deletion
+// won't be incremented.
+XAttr existingXAttr = FSDirXAttrOp.getXAttrByPrefixedName(
+fsdir, INodesInPath.append(iip, snapshot.getRoot(),
Review comment:
we can use snaphot.getRoot().isMarkedAsDeleted() here instead?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
##
@@ -459,4 +472,23 @@ public void addEditLogTailInterval(long elapsed) {
q.add(elapsed);
}
}
+ public void incrNumSnapshotDelete() {
+numSnapshotTotalDeleteOp.incr();
+ }
+ public void incrOutOfOrderSnapshotDelete() {
+numSnapshotOutOfOrderDeleteOp.incr();
+ }
+ public void incrInOrderSnapshotDelete() {
+numSnapshotInOrderDeleteOp.incr();
+ }
Review comment:
its better to move these 2 metrics to SnapshotMetrcics.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1839: HADOOP-16848. Refactoring: initial layering
hadoop-yetus commented on pull request #1839:
URL: https://github.com/apache/hadoop/pull/1839#issuecomment-770969437
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 7s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 5s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -0 :warning: | checkstyle | 0m 22s |
[/diff-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 37 new + 42 unchanged - 0
fixed = 79 total (was 42) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/whitespace-eol.txt)
| The patch has 2 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 12m 38s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javadoc | 0m 27s |
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 7 new +
88 unchanged - 0 fixed = 95 total (was 88) |
| -1 :x: | findbugs | 1m 8s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total
(was 0) |
_ Other Tests _ |
| -1 :x: | unit | 1m 58s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 73m 42s | | |
| Reason | Tests |
|---:|:--|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AFileSystem.signerManager; locked 66% of time
Unsynchronized access at S3AFileSystem.java:66% of time Unsynchronized access
at S3AFileSystem.java:[line 4864] |
| Failed junit tests | hadoop.fs.s3a.TestS3ABlockOutputStream |
| | hadoop.fs.s3a.s3guard.TestObjectChangeDetectionAttributes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1
[jira] [Work logged] (HADOOP-16848) add initial S3A layering + async init
[
https://issues.apache.org/jira/browse/HADOOP-16848?focusedWorklogId=545418&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545418
]
ASF GitHub Bot logged work on HADOOP-16848:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 16:10
Start Date: 01/Feb/21 16:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #1839:
URL: https://github.com/apache/hadoop/pull/1839#issuecomment-770969437
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 7s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 5s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -0 :warning: | checkstyle | 0m 22s |
[/diff-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 37 new + 42 unchanged - 0
fixed = 79 total (was 42) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/whitespace-eol.txt)
| The patch has 2 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 12m 38s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javadoc | 0m 27s |
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 7 new +
88 unchanged - 0 fixed = 95 total (was 88) |
| -1 :x: | findbugs | 1m 8s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total
(was 0) |
_ Other Tests _ |
| -1 :x: | unit | 1m 58s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1839/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 73m 42s | | |
| Reason | Tests |
|---:|:--|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Inconsistent synchronization of
org.ap
[GitHub] [hadoop] hadoop-yetus commented on pull request #2664: HADOOP-17508. Simplify deps install instructions
hadoop-yetus commented on pull request #2664:
URL: https://github.com/apache/hadoop/pull/2664#issuecomment-770974537
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 20m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 12m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 36m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2664/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2664 |
| Optional Tests | dupname asflicense |
| uname | Linux 4a36d76055ab 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b54134661b5 |
| Max. process+thread count | 746 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2664/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17508) Simplify dependency installation instructions
[
https://issues.apache.org/jira/browse/HADOOP-17508?focusedWorklogId=545423&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545423
]
ASF GitHub Bot logged work on HADOOP-17508:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 16:17
Start Date: 01/Feb/21 16:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2664:
URL: https://github.com/apache/hadoop/pull/2664#issuecomment-770974537
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 20m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 12m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 36m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2664/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2664 |
| Optional Tests | dupname asflicense |
| uname | Linux 4a36d76055ab 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b54134661b5 |
| Max. process+thread count | 746 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2664/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545423)
Time Spent: 1h 10m (was: 1h)
> Simplify dependency installation instructions
> -
>
> Key: HADOOP-17508
> URL: https://issues.apache.org/jira/browse/HADOOP-17508
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The instructions for installing Protocol buffers and Boost seem to just be
> copied from Dockerfile and isn't friendly to read.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17476) ITestAssumeRole.testAssumeRoleBadInnerAuth failure
[
https://issues.apache.org/jira/browse/HADOOP-17476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276454#comment-17276454
]
Steve Loughran commented on HADOOP-17476:
-
It's back, at least on the HADOOP-16202 enhance-openfile branch. Key is error
code IncompleteSignature
> ITestAssumeRole.testAssumeRoleBadInnerAuth failure
> --
>
> Key: HADOOP-17476
> URL: https://issues.apache.org/jira/browse/HADOOP-17476
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Priority: Minor
>
> failure of test {{ITestAssumeRole.testAssumeRoleBadInnerAuth}} where a
> failure was expected, but the error text was wrong.
> Either STS has changed its error text or something is changing where the
> failure happens.
> Given the nature of the test, it may be simplest to keep the expectation of
> an FS init faiure, but remove the text match
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17476) ITestAssumeRole.testAssumeRoleBadInnerAuth failure
[
https://issues.apache.org/jira/browse/HADOOP-17476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276458#comment-17276458
]
Steve Loughran commented on HADOOP-17476:
-
that's with test options of {{-Dparallel-tests -DtestsThreadCount=6
-Dmarkers=keep}}
.i.e. no s3guard. And the exception returned is
{{AWSSecurityTokenServiceException}}. Maybe on an unguarded bucket the STS
exception is raised without translation.
Anyway, it's still a form of (expected) failuretest needs relaxing
> ITestAssumeRole.testAssumeRoleBadInnerAuth failure
> --
>
> Key: HADOOP-17476
> URL: https://issues.apache.org/jira/browse/HADOOP-17476
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Priority: Minor
>
> failure of test {{ITestAssumeRole.testAssumeRoleBadInnerAuth}} where a
> failure was expected, but the error text was wrong.
> Either STS has changed its error text or something is changing where the
> failure happens.
> Given the nature of the test, it may be simplest to keep the expectation of
> an FS init faiure, but remove the text match
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2669: HADOOP-17509. Parallilize building of deps
hadoop-yetus commented on pull request #2669:
URL: https://github.com/apache/hadoop/pull/2669#issuecomment-771004420
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 18s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 23m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| -1 :x: | hadolint | 0m 4s |
[/diff-patch-hadolint.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/artifact/out/diff-patch-hadolint.txt)
| The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new
shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 15s | | There were no new
shelldocs issues. |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 20m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 72m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2669 |
| Optional Tests | dupname asflicense hadolint shellcheck shelldocs |
| uname | Linux e334271442f7 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b54134661b5 |
| Max. process+thread count | 572 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17509) Parallelize building of dependencies
[
https://issues.apache.org/jira/browse/HADOOP-17509?focusedWorklogId=545447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545447
]
ASF GitHub Bot logged work on HADOOP-17509:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 17:01
Start Date: 01/Feb/21 17:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2669:
URL: https://github.com/apache/hadoop/pull/2669#issuecomment-771004420
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 18s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 23m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| -1 :x: | hadolint | 0m 4s |
[/diff-patch-hadolint.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/artifact/out/diff-patch-hadolint.txt)
| The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new
shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 15s | | There were no new
shelldocs issues. |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 20m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 72m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2669 |
| Optional Tests | dupname asflicense hadolint shellcheck shelldocs |
| uname | Linux e334271442f7 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b54134661b5 |
| Max. process+thread count | 572 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545447)
Time Spent: 0.5h (was: 20m)
> Parallelize building of dependencies
>
>
> Key: HADOOP-17509
> URL: https://issues.apache.org/jira/browse/HADOOP-17509
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.3.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Need to use make -j$(nproc) to parallelize building of Protocol buffers and
> Intel ISA - L dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2548: HADOOP-17475. ABFS: Implementing ListStatusRemoteIterator
steveloughran commented on a change in pull request #2548:
URL: https://github.com/apache/hadoop/pull/2548#discussion_r567989510
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsListStatusRemoteIterator.java
##
@@ -0,0 +1,338 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.azurebfs.services.AbfsListStatusRemoteIterator;
+import org.apache.hadoop.fs.azurebfs.services.ListingSupport;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Mockito.verify;
+
+/**
+ * Test ListStatusRemoteIterator operation.
+ */
+public class ITestAbfsListStatusRemoteIterator extends
AbstractAbfsIntegrationTest {
+
+ private static final int TEST_FILES_NUMBER = 1000;
+
+ public ITestAbfsListStatusRemoteIterator() throws Exception {
+super();
Review comment:
cut
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##
@@ -982,6 +985,19 @@ public boolean exists(Path f) throws IOException {
return super.exists(f);
}
+ @Override
+ public RemoteIterator listStatusIterator(Path path)
+ throws IOException {
+LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
+if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ AbfsListStatusRemoteIterator abfsLsItr =
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ return RemoteIterators.typeCastingRemoteIterator(abfsLsItr);
Review comment:
nice
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsListStatusRemoteIterator.java
##
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import javax.activation.UnsupportedDataTypeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.RemoteIterator;
+
+public class AbfsListStatusRemoteIterator
+implements RemoteIterator {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(AbfsListStatusRemoteIterator.class);
+
+ private static final boolean FETCH_ALL_FALSE = false;
+ private static final int MA
[jira] [Work logged] (HADOOP-17475) Implement listStatusIterator
[
https://issues.apache.org/jira/browse/HADOOP-17475?focusedWorklogId=545456&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545456
]
ASF GitHub Bot logged work on HADOOP-17475:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 17:17
Start Date: 01/Feb/21 17:17
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2548:
URL: https://github.com/apache/hadoop/pull/2548#discussion_r567989510
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsListStatusRemoteIterator.java
##
@@ -0,0 +1,338 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.azurebfs.services.AbfsListStatusRemoteIterator;
+import org.apache.hadoop.fs.azurebfs.services.ListingSupport;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Mockito.verify;
+
+/**
+ * Test ListStatusRemoteIterator operation.
+ */
+public class ITestAbfsListStatusRemoteIterator extends
AbstractAbfsIntegrationTest {
+
+ private static final int TEST_FILES_NUMBER = 1000;
+
+ public ITestAbfsListStatusRemoteIterator() throws Exception {
+super();
Review comment:
cut
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##
@@ -982,6 +985,19 @@ public boolean exists(Path f) throws IOException {
return super.exists(f);
}
+ @Override
+ public RemoteIterator listStatusIterator(Path path)
+ throws IOException {
+LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
+if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ AbfsListStatusRemoteIterator abfsLsItr =
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ return RemoteIterators.typeCastingRemoteIterator(abfsLsItr);
Review comment:
nice
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsListStatusRemoteIterator.java
##
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import javax.activation.UnsupportedDataTypeE
[GitHub] [hadoop] hadoop-yetus commented on pull request #2668: HDFS-15808. Add metrics for FSNamesystem read/write lock warnings
hadoop-yetus commented on pull request #2668:
URL: https://github.com/apache/hadoop/pull/2668#issuecomment-771020821
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 3m 19s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| -0 :warning: | checkstyle | 1m 1s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2668/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 165 unchanged
- 0 fixed = 173 total (was 165) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 21s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 3m 18s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 193m 50s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2668/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 281m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
|
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2668/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2668 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a123314fa64f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 115623a6ee4 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2668/1/testReport/ |
| Max. process+thread count | 3
[GitHub] [hadoop] goiri merged pull request #2633: HADOOP-17482: Remove Commons Logger from FileSystem Class
goiri merged pull request #2633: URL: https://github.com/apache/hadoop/pull/2633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[ https://issues.apache.org/jira/browse/HADOOP-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-17482: - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Remove Commons Logger from FileSystem Class > --- > > Key: HADOOP-17482 > URL: https://issues.apache.org/jira/browse/HADOOP-17482 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Remove reference to Commons Logger in FileSystem, it already has SLF4J, so > it's a bit weird to be mixing and matching and interweaving loggers in this > way. Also, my hope is to eventually migrate everything to SLF4J to simplify > things for downstream consumers of the common library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[ https://issues.apache.org/jira/browse/HADOOP-17482?focusedWorklogId=545473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545473 ] ASF GitHub Bot logged work on HADOOP-17482: --- Author: ASF GitHub Bot Created on: 01/Feb/21 17:40 Start Date: 01/Feb/21 17:40 Worklog Time Spent: 10m Work Description: goiri merged pull request #2633: URL: https://github.com/apache/hadoop/pull/2633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545473) Time Spent: 4h (was: 3h 50m) > Remove Commons Logger from FileSystem Class > --- > > Key: HADOOP-17482 > URL: https://issues.apache.org/jira/browse/HADOOP-17482 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > Remove reference to Commons Logger in FileSystem, it already has SLF4J, so > it's a bit weird to be mixing and matching and interweaving loggers in this > way. Also, my hope is to eventually migrate everything to SLF4J to simplify > things for downstream consumers of the common library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[ https://issues.apache.org/jira/browse/HADOOP-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276527#comment-17276527 ] Íñigo Goiri commented on HADOOP-17482: -- Merged PR 2633. Thanks [~belugabehr] for the contribution. > Remove Commons Logger from FileSystem Class > --- > > Key: HADOOP-17482 > URL: https://issues.apache.org/jira/browse/HADOOP-17482 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Remove reference to Commons Logger in FileSystem, it already has SLF4J, so > it's a bit weird to be mixing and matching and interweaving loggers in this > way. Also, my hope is to eventually migrate everything to SLF4J to simplify > things for downstream consumers of the common library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17510) Hadoop prints sensitive Cookie information.
Renukaprasad C created HADOOP-17510:
---
Summary: Hadoop prints sensitive Cookie information.
Key: HADOOP-17510
URL: https://issues.apache.org/jira/browse/HADOOP-17510
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Renukaprasad C
org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
- prints cookie information in log. Any sensitive infomation in Cookies will
be logged, which needs to be avaided.
LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renukaprasad C updated HADOOP-17510:
Attachment: HDFS-15805.001.patch
Status: Patch Available (was: Open)
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HDFS-15805.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276551#comment-17276551
]
Renukaprasad C commented on HADOOP-17510:
-
Log removal, no new test case added. No new test failures.
[~hemanthboyina] Please review when you get time. Thank you.
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HDFS-15805.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2633: HADOOP-17482: Remove Commons Logger from FileSystem Class
steveloughran commented on a change in pull request #2633:
URL: https://github.com/apache/hadoop/pull/2633#discussion_r568035635
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java
##
@@ -100,16 +97,6 @@ public static void ClusterShutdownAtEnd() throws Exception {
cluster.shutdown();
}
}
-
- {
-try {
- GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
-}
-catch(Exception e) {
- System.out.println("Cannot change log level\n"
- + StringUtils.stringifyException(e));
-}
- }
Review comment:
its a test and they may have had some good reasons, just as some of the
Azure tests do
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[
https://issues.apache.org/jira/browse/HADOOP-17482?focusedWorklogId=545489&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545489
]
ASF GitHub Bot logged work on HADOOP-17482:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 18:13
Start Date: 01/Feb/21 18:13
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2633:
URL: https://github.com/apache/hadoop/pull/2633#discussion_r568035635
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java
##
@@ -100,16 +97,6 @@ public static void ClusterShutdownAtEnd() throws Exception {
cluster.shutdown();
}
}
-
- {
-try {
- GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
-}
-catch(Exception e) {
- System.out.println("Cannot change log level\n"
- + StringUtils.stringifyException(e));
-}
- }
Review comment:
its a test and they may have had some good reasons, just as some of the
Azure tests do
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545489)
Time Spent: 4h 10m (was: 4h)
> Remove Commons Logger from FileSystem Class
> ---
>
> Key: HADOOP-17482
> URL: https://issues.apache.org/jira/browse/HADOOP-17482
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Remove reference to Commons Logger in FileSystem, it already has SLF4J, so
> it's a bit weird to be mixing and matching and interweaving loggers in this
> way. Also, my hope is to eventually migrate everything to SLF4J to simplify
> things for downstream consumers of the common library.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[
https://issues.apache.org/jira/browse/HADOOP-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276554#comment-17276554
]
Steve Loughran commented on HADOOP-17482:
-
ok, there's a story here. The interface was protected and we didn't know which
external subclasses, if any, were using it.
This is why the comment above the line you've just changed says
{code}
/**
* This log is widely used in the org.apache.hadoop.fs code and tests,
* so must be considered something to only be changed with care.
*/
{code}
I see you've discovered most of the issues there. There's one more: a risk of
external use. Can you add it to the release notes
> Remove Commons Logger from FileSystem Class
> ---
>
> Key: HADOOP-17482
> URL: https://issues.apache.org/jira/browse/HADOOP-17482
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Remove reference to Commons Logger in FileSystem, it already has SLF4J, so
> it's a bit weird to be mixing and matching and interweaving loggers in this
> way. Also, my hope is to eventually migrate everything to SLF4J to simplify
> things for downstream consumers of the common library.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2633: HADOOP-17482: Remove Commons Logger from FileSystem Class
goiri commented on a change in pull request #2633:
URL: https://github.com/apache/hadoop/pull/2633#discussion_r568038118
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java
##
@@ -100,16 +97,6 @@ public static void ClusterShutdownAtEnd() throws Exception {
cluster.shutdown();
}
}
-
- {
-try {
- GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
-}
-catch(Exception e) {
- System.out.println("Cannot change log level\n"
- + StringUtils.stringifyException(e));
-}
- }
Review comment:
We need to fix it properly then. @belugabehr do you mind undoing this
particular one in a separate JIRA or proposing the log4j solution?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[
https://issues.apache.org/jira/browse/HADOOP-17482?focusedWorklogId=545492&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545492
]
ASF GitHub Bot logged work on HADOOP-17482:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 18:17
Start Date: 01/Feb/21 18:17
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2633:
URL: https://github.com/apache/hadoop/pull/2633#discussion_r568038118
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java
##
@@ -100,16 +97,6 @@ public static void ClusterShutdownAtEnd() throws Exception {
cluster.shutdown();
}
}
-
- {
-try {
- GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
-}
-catch(Exception e) {
- System.out.println("Cannot change log level\n"
- + StringUtils.stringifyException(e));
-}
- }
Review comment:
We need to fix it properly then. @belugabehr do you mind undoing this
particular one in a separate JIRA or proposing the log4j solution?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545492)
Time Spent: 4h 20m (was: 4h 10m)
> Remove Commons Logger from FileSystem Class
> ---
>
> Key: HADOOP-17482
> URL: https://issues.apache.org/jira/browse/HADOOP-17482
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Remove reference to Commons Logger in FileSystem, it already has SLF4J, so
> it's a bit weird to be mixing and matching and interweaving loggers in this
> way. Also, my hope is to eventually migrate everything to SLF4J to simplify
> things for downstream consumers of the common library.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renukaprasad C updated HADOOP-17510:
Attachment: HADOOP-17510.001.patch
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HADOOP-17510.001.patch, HDFS-15805.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renukaprasad C updated HADOOP-17510:
Attachment: (was: HDFS-15805.001.patch)
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HADOOP-17510.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2584: HADOOP-16202. Enhance openFile()
steveloughran commented on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771070179 rebased to fix compile problems; the final patch is the one with changes since Thomas's last review * move all the text on options into the fsdatainputstreambuilder.md file * try to define the validation of must/opt in pseudo-python * make sure the doc is consistent in saying FNFE and permission issues MUST be delayed until future.get() and MAY be delayed until the first actual read. This is to give object stores the maximum time for async probes. Example: you could initiate a HEAD request in build() but not block for its completion until read() S3A added some extra tuning of the input stream/openFile logic, so the stream no longer gets access to the S3 client, simply callbacks Testing: in progress This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276573#comment-17276573
]
Steve Loughran commented on HADOOP-17510:
-
can you stick in as a github PR
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HADOOP-17510.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=545505&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545505
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 18:40
Start Date: 01/Feb/21 18:40
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771070179
rebased to fix compile problems; the final patch is the one with changes
since Thomas's last review
* move all the text on options into the fsdatainputstreambuilder.md file
* try to define the validation of must/opt in pseudo-python
* make sure the doc is consistent in saying FNFE and permission issues MUST
be delayed until future.get() and MAY be delayed until the first actual read.
This is to give object stores the maximum time for async probes. Example: you
could initiate a HEAD request in build() but not block for its completion
until read()
S3A added some extra tuning of the input stream/openFile logic, so the
stream no longer gets access to the S3 client, simply callbacks
Testing: in progress
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545505)
Time Spent: 9h 20m (was: 9h 10m)
> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276573#comment-17276573
]
Steve Loughran edited comment on HADOOP-17510 at 2/1/21, 6:41 PM:
--
can you submit in as a github PR? Thanks
was (Author: ste...@apache.org):
can you stick in as a github PR
> Hadoop prints sensitive Cookie information.
> ---
>
> Key: HADOOP-17510
> URL: https://issues.apache.org/jira/browse/HADOOP-17510
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Renukaprasad C
>Priority: Major
> Attachments: HADOOP-17510.001.patch
>
>
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.AuthCookieHandler#setAuthCookie
> - prints cookie information in log. Any sensitive infomation in Cookies will
> be logged, which needs to be avaided.
> LOG.trace("Setting token value to {} ({})", authCookie, oldCookie);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on a change in pull request #2664: HADOOP-17508. Simplify deps install instructions
smengcl commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r568072676 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src + $ ./configure + $ make -j$(nproc) Review comment: Sure. Will do :D This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17508) Simplify dependency installation instructions
[ https://issues.apache.org/jira/browse/HADOOP-17508?focusedWorklogId=545518&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545518 ] ASF GitHub Bot logged work on HADOOP-17508: --- Author: ASF GitHub Bot Created on: 01/Feb/21 19:11 Start Date: 01/Feb/21 19:11 Worklog Time Spent: 10m Work Description: smengcl commented on a change in pull request #2664: URL: https://github.com/apache/hadoop/pull/2664#discussion_r568072676 ## File path: BUILDING.txt ## @@ -76,21 +76,16 @@ Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: $ make -j$(nproc) $ sudo make install * Protocol Buffers 3.7.1 (required to build native code) - $ mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure\ -&& make install \ -&& rm -rf /opt/protobuf-3.7-src + $ curl -L -s -S https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz -o protobuf-3.7.1.tar.gz + $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src && cd protobuf-3.7-src + $ ./configure + $ make -j$(nproc) Review comment: Sure. Will do :D This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545518) Time Spent: 1h 20m (was: 1h 10m) > Simplify dependency installation instructions > - > > Key: HADOOP-17508 > URL: https://issues.apache.org/jira/browse/HADOOP-17508 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Trivial > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > The instructions for installing Protocol buffers and Boost seem to just be > copied from Dockerfile and isn't friendly to read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276647#comment-17276647
]
Hadoop QA commented on HADOOP-17510:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 42s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
52s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
19s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
19s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 47s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.
[jira] [Commented] (HADOOP-17510) Hadoop prints sensitive Cookie information.
[
https://issues.apache.org/jira/browse/HADOOP-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276661#comment-17276661
]
Hadoop QA commented on HADOOP-17510:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 17s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
56s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
33s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 36s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.
[GitHub] [hadoop] symious commented on pull request #2659: HDFS-15801. Backport HDFS-14582 to branch-2.10 (Failed to start DN wi…
symious commented on pull request #2659: URL: https://github.com/apache/hadoop/pull/2659#issuecomment-771212244 @jojochuang Thanks for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zehaoc2 opened a new pull request #2670: HDFS-15811. completeFile should log final file size.
zehaoc2 opened a new pull request #2670: URL: https://github.com/apache/hadoop/pull/2670 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2584: HADOOP-16202. Enhance openFile()
hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771228069
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 20m 45s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 6s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 33s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 54s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 46s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 16s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 53s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 21s | | the patch passed |
| +1 :green_heart: | compile | 20m 8s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 8s | | the patch passed |
| +1 :green_heart: | compile | 18m 3s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 18m 3s | | the patch passed |
| -0 :warning: | checkstyle | 4m 0s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 821 unchanged - 1 fixed = 826 total (was
822) |
| +1 :green_heart: | mvnsite | 7m 22s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/whitespace-eol.txt)
| The patch has 10 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 13m 12s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 48s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 44s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 11m 59s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 54s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 43s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 48s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 9m 49s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | unit | 15m 45s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | unit | 1m 9s | | hadoop-mapreduce-examples in
the patch passed. |
| -1 :x: | unit | 6m 46s |
[/patch-unit-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-unit-hadoop-tools_hadoop-streaming.txt)
| hadoop-streaming in the patch passed. |
| +1 :green_heart: | unit | 2m 34s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 288m 49s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.streaming.TestSymLink |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/Dockerfile
|
| GITHUB PR | https://gi
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=545690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545690
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 01/Feb/21 23:17
Start Date: 01/Feb/21 23:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-771228069
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 20m 45s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 6s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 33s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 54s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 46s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +0 :ok: | spotbugs | 1m 16s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 53s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 21s | | the patch passed |
| +1 :green_heart: | compile | 20m 8s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 8s | | the patch passed |
| +1 :green_heart: | compile | 18m 3s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | javac | 18m 3s | | the patch passed |
| -0 :warning: | checkstyle | 4m 0s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 821 unchanged - 1 fixed = 826 total (was
822) |
| +1 :green_heart: | mvnsite | 7m 22s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/whitespace-eol.txt)
| The patch has 10 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 13m 12s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 48s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 6m 44s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
| +1 :green_heart: | findbugs | 11m 59s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 54s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 43s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 48s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 9m 49s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | unit | 15m 45s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | unit | 1m 9s | | hadoop-mapreduce-examples in
the patch passed. |
| -1 :x: | unit | 6m 46s |
[/patch-unit-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/1/artifact/out/patch-unit-hadoop-tools_hadoop-streaming.txt)
| hadoop-streaming in the patch passed. |
| +1 :green_heart: | unit | 2m 34s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 1s | | Th
[GitHub] [hadoop] jojochuang merged pull request #2659: HDFS-15801. Backport HDFS-14582 to branch-2.10 (Failed to start DN wi…
jojochuang merged pull request #2659: URL: https://github.com/apache/hadoop/pull/2659 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] theRealELiu commented on pull request #2546: HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
theRealELiu commented on pull request #2546: URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771303789 Have you tried that if ``` export YARN_OPTS="--illegal-access=permit" export YARN_OPTS="--illegal-access=permit" ``` could solve the problem as a workaround? @aajisaka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17432) [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
[
https://issues.apache.org/jira/browse/HADOOP-17432?focusedWorklogId=545738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545738
]
ASF GitHub Bot logged work on HADOOP-17432:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 02:29
Start Date: 02/Feb/21 02:29
Worklog Time Spent: 10m
Work Description: theRealELiu commented on pull request #2546:
URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771303789
Have you tried that if
```
export YARN_OPTS="--illegal-access=permit"
export YARN_OPTS="--illegal-access=permit"
```
could solve the problem as a workaround?
@aajisaka
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545738)
Time Spent: 40m (was: 0.5h)
> [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
> -
>
> Key: HADOOP-17432
> URL: https://issues.apache.org/jira/browse/HADOOP-17432
> Project: Hadoop Common
> Issue Type: Sub-task
> Environment: Java 16-ea+28
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> JEP 396 (Strongly Encapsulate JDK Internals by Default) has been migrated
> since Java 16 EA Build 28. Calling the internal APIs (except the critical
> APIs such as sun.misc.Unsafe) are banned by default.
> {noformat}
> [INFO] Running
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR] Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 17.323 s <<< FAILURE! - in
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR]
> testNonDefaultNonBrowserUserAgentAsNonBrowser(org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler)
> Time elapsed: 1.272 s <<< ERROR!
> java.lang.IllegalAccessException: class
> org.apache.hadoop.security.authentication.util.KerberosUtil cannot access
> class sun.security.jgss.GSSUtil (in module java.security.jgss) because module
> java.security.jgss does not export sun.security.jgss to unnamed module
> @48967c8b
> at
> java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)
> at
> java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)
> at java.base/java.lang.reflect.Field.checkAccess(Field.java:1096)
> at java.base/java.lang.reflect.Field.get(Field.java:417)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.getOidInstance(KerberosUtil.java:90)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] theRealELiu edited a comment on pull request #2546: HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
theRealELiu edited a comment on pull request #2546: URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771303789 Have you tried that if ``` export HADOOP_OPTS="--illegal-access=permit" export YARN_OPTS="--illegal-access=permit" ``` could solve the problem as a workaround? @aajisaka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17432) [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
[
https://issues.apache.org/jira/browse/HADOOP-17432?focusedWorklogId=545743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545743
]
ASF GitHub Bot logged work on HADOOP-17432:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 02:39
Start Date: 02/Feb/21 02:39
Worklog Time Spent: 10m
Work Description: theRealELiu edited a comment on pull request #2546:
URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771303789
Have you tried that if
```
export HADOOP_OPTS="--illegal-access=permit"
export YARN_OPTS="--illegal-access=permit"
```
could solve the problem as a workaround?
@aajisaka
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545743)
Time Spent: 50m (was: 40m)
> [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
> -
>
> Key: HADOOP-17432
> URL: https://issues.apache.org/jira/browse/HADOOP-17432
> Project: Hadoop Common
> Issue Type: Sub-task
> Environment: Java 16-ea+28
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> JEP 396 (Strongly Encapsulate JDK Internals by Default) has been migrated
> since Java 16 EA Build 28. Calling the internal APIs (except the critical
> APIs such as sun.misc.Unsafe) are banned by default.
> {noformat}
> [INFO] Running
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR] Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 17.323 s <<< FAILURE! - in
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR]
> testNonDefaultNonBrowserUserAgentAsNonBrowser(org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler)
> Time elapsed: 1.272 s <<< ERROR!
> java.lang.IllegalAccessException: class
> org.apache.hadoop.security.authentication.util.KerberosUtil cannot access
> class sun.security.jgss.GSSUtil (in module java.security.jgss) because module
> java.security.jgss does not export sun.security.jgss to unnamed module
> @48967c8b
> at
> java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)
> at
> java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)
> at java.base/java.lang.reflect.Field.checkAccess(Field.java:1096)
> at java.base/java.lang.reflect.Field.get(Field.java:417)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.getOidInstance(KerberosUtil.java:90)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HeartSaVioR commented on pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
HeartSaVioR commented on pull request #2667: URL: https://github.com/apache/hadoop/pull/2667#issuecomment-771308533 Addressed the basic UT, IT, scaled IT. Once the Yetus is happy with the change, I'll remove WIP and ping again for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545744
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 02:42
Start Date: 02/Feb/21 02:42
Worklog Time Spent: 10m
Work Description: HeartSaVioR commented on pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#issuecomment-771308533
Addressed the basic UT, IT, scaled IT. Once the Yetus is happy with the
change, I'll remove WIP and ping again for review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545744)
Time Spent: 50m (was: 40m)
> Add some Abortable.abort() interface for streams etc which can be terminated
>
>
> Key: HADOOP-16906
> URL: https://issues.apache.org/jira/browse/HADOOP-16906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some IO we want to be able to abort rather than close cleanly, especially if
> the inner stream is an HTTP connection which itself supports some abort()
> method. For example: uploads to an object where we want to cancel the upload
> without close() making an incomplete write visible.
> Proposed: Add a generic interface which things like streams can implement
> {code}
> AbortableIO {
> public void abortIO() throws IOE;
> }
> {code}
> +do for s3a output stream. I wouldn't do this a passthrough on
> FSDataOutputStream because we need to consider what expectations callers have
> of an operation being "aborted"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2546: HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
aajisaka commented on pull request #2546: URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771309651 @theRealELiu I think the workaround works. However, it will force users/admins to set the `--illegal-access=permit` option to all the Hadoop applications, which is not ideal. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17432) [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
[
https://issues.apache.org/jira/browse/HADOOP-17432?focusedWorklogId=545746&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545746
]
ASF GitHub Bot logged work on HADOOP-17432:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 02:45
Start Date: 02/Feb/21 02:45
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #2546:
URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771309651
@theRealELiu
I think the workaround works. However, it will force users/admins to set the
`--illegal-access=permit` option to all the Hadoop applications, which is not
ideal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545746)
Time Spent: 1h (was: 50m)
> [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
> -
>
> Key: HADOOP-17432
> URL: https://issues.apache.org/jira/browse/HADOOP-17432
> Project: Hadoop Common
> Issue Type: Sub-task
> Environment: Java 16-ea+28
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JEP 396 (Strongly Encapsulate JDK Internals by Default) has been migrated
> since Java 16 EA Build 28. Calling the internal APIs (except the critical
> APIs such as sun.misc.Unsafe) are banned by default.
> {noformat}
> [INFO] Running
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR] Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 17.323 s <<< FAILURE! - in
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR]
> testNonDefaultNonBrowserUserAgentAsNonBrowser(org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler)
> Time elapsed: 1.272 s <<< ERROR!
> java.lang.IllegalAccessException: class
> org.apache.hadoop.security.authentication.util.KerberosUtil cannot access
> class sun.security.jgss.GSSUtil (in module java.security.jgss) because module
> java.security.jgss does not export sun.security.jgss to unnamed module
> @48967c8b
> at
> java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)
> at
> java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)
> at java.base/java.lang.reflect.Field.checkAccess(Field.java:1096)
> at java.base/java.lang.reflect.Field.get(Field.java:417)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.getOidInstance(KerberosUtil.java:90)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HeartSaVioR commented on pull request #2667: WIP. HADOOP-16906. Add Abortable.abort() interface for streams to enable output stream to be terminated
HeartSaVioR commented on pull request #2667: URL: https://github.com/apache/hadoop/pull/2667#issuecomment-771310932 I don't get the warning sign findbugs provided; `multipartUpload` field is already accessed via both synchronized/unsynchronized ways before this change, so probably something to ignore here? https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16906) Add some Abortable.abort() interface for streams etc which can be terminated
[
https://issues.apache.org/jira/browse/HADOOP-16906?focusedWorklogId=545750&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545750
]
ASF GitHub Bot logged work on HADOOP-16906:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 02:48
Start Date: 02/Feb/21 02:48
Worklog Time Spent: 10m
Work Description: HeartSaVioR commented on pull request #2667:
URL: https://github.com/apache/hadoop/pull/2667#issuecomment-771310932
I don't get the warning sign findbugs provided; `multipartUpload` field is
already accessed via both synchronized/unsynchronized ways before this change,
so probably something to ignore here?
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2667/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545750)
Time Spent: 1h (was: 50m)
> Add some Abortable.abort() interface for streams etc which can be terminated
>
>
> Key: HADOOP-16906
> URL: https://issues.apache.org/jira/browse/HADOOP-16906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Some IO we want to be able to abort rather than close cleanly, especially if
> the inner stream is an HTTP connection which itself supports some abort()
> method. For example: uploads to an object where we want to cancel the upload
> without close() making an incomplete write visible.
> Proposed: Add a generic interface which things like streams can implement
> {code}
> AbortableIO {
> public void abortIO() throws IOE;
> }
> {code}
> +do for s3a output stream. I wouldn't do this a passthrough on
> FSDataOutputStream because we need to consider what expectations callers have
> of an operation being "aborted"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] theRealELiu commented on pull request #2546: HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
theRealELiu commented on pull request #2546: URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771318559 Thanks for your explanation. Do you know the release date of `3.3.1` or `3.4`? And is there any plan to support java-17, it's a LTS version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17432) [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
[
https://issues.apache.org/jira/browse/HADOOP-17432?focusedWorklogId=545759&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545759
]
ASF GitHub Bot logged work on HADOOP-17432:
---
Author: ASF GitHub Bot
Created on: 02/Feb/21 03:07
Start Date: 02/Feb/21 03:07
Worklog Time Spent: 10m
Work Description: theRealELiu commented on pull request #2546:
URL: https://github.com/apache/hadoop/pull/2546#issuecomment-771318559
Thanks for your explanation.
Do you know the release date of `3.3.1` or `3.4`? And is there any plan to
support java-17, it's a LTS version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 545759)
Time Spent: 1h 10m (was: 1h)
> [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396
> -
>
> Key: HADOOP-17432
> URL: https://issues.apache.org/jira/browse/HADOOP-17432
> Project: Hadoop Common
> Issue Type: Sub-task
> Environment: Java 16-ea+28
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> JEP 396 (Strongly Encapsulate JDK Internals by Default) has been migrated
> since Java 16 EA Build 28. Calling the internal APIs (except the critical
> APIs such as sun.misc.Unsafe) are banned by default.
> {noformat}
> [INFO] Running
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR] Tests run: 16, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 17.323 s <<< FAILURE! - in
> org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
> [ERROR]
> testNonDefaultNonBrowserUserAgentAsNonBrowser(org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler)
> Time elapsed: 1.272 s <<< ERROR!
> java.lang.IllegalAccessException: class
> org.apache.hadoop.security.authentication.util.KerberosUtil cannot access
> class sun.security.jgss.GSSUtil (in module java.security.jgss) because module
> java.security.jgss does not export sun.security.jgss to unnamed module
> @48967c8b
> at
> java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)
> at
> java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)
> at java.base/java.lang.reflect.Field.checkAccess(Field.java:1096)
> at java.base/java.lang.reflect.Field.get(Field.java:417)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.getOidInstance(KerberosUtil.java:90)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2651: HDFS-15757 RBF: Improving Router Connection Management
goiri commented on a change in pull request #2651: URL: https://github.com/apache/hadoop/pull/2651#discussion_r568315552 ## File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionContext.java ## @@ -42,7 +44,10 @@ private int numThreads = 0; /** If the connection is closed. */ private boolean closed = false; - + /** Last timestamp the connection was active. */ + private long lastActiveTs = 0; + /** The connection's active status would expire after this window. */ + private long activeWindow = TimeUnit.SECONDS.toMillis(30); Review comment: I agree. and called ACTIVE_WINDOW_TIME This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2651: HDFS-15757 RBF: Improving Router Connection Management
goiri commented on a change in pull request #2651:
URL: https://github.com/apache/hadoop/pull/2651#discussion_r568315959
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionContext.java
##
@@ -57,6 +62,17 @@ public synchronized boolean isActive() {
return this.numThreads > 0;
}
+ /**
+ * Check if the connection is/was active recently.
+ *
+ * @return True if the connection is active or
+ * was active in the past period of time.
+ */
+ public synchronized boolean isActiveRecently() {
+return isActive() ||
Review comment:
If we do the timestamp based, do we even need to check for the
isActive() or the timestamp comparisson is enoug?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims merged pull request #2664: HADOOP-17508. Simplify deps install instructions
iwasakims merged pull request #2664: URL: https://github.com/apache/hadoop/pull/2664 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17508) Simplify dependency installation instructions
[ https://issues.apache.org/jira/browse/HADOOP-17508?focusedWorklogId=545797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-545797 ] ASF GitHub Bot logged work on HADOOP-17508: --- Author: ASF GitHub Bot Created on: 02/Feb/21 05:33 Start Date: 02/Feb/21 05:33 Worklog Time Spent: 10m Work Description: iwasakims merged pull request #2664: URL: https://github.com/apache/hadoop/pull/2664 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 545797) Time Spent: 1.5h (was: 1h 20m) > Simplify dependency installation instructions > - > > Key: HADOOP-17508 > URL: https://issues.apache.org/jira/browse/HADOOP-17508 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Trivial > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The instructions for installing Protocol buffers and Boost seem to just be > copied from Dockerfile and isn't friendly to read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
